Publishing reduced spinner file changes to a git repo

Our group faces a unique challenge based on the way we operate.

To spin in changes to the 3DX database, we use reduced versions of the spinner files that only contain only our related changes. This works because the framework can process added/changed/deleted rows separately without requiring the entire rest of the file (i.e.: if the master SpinnerWebFormFieldData_ALL.xls file contains 20000+ rows, but we only need to add 1 new one to show a new field on a form, so we create a SpinnerWebFormFieldData_DELTA.xls with just the header row and the new row to spin in the changes).

The biggest benefit to spinning in changes this way is drastically reduced spin time (i.e.: for comparison, that same file might have taken 15 minutes to process that one file once while the reduced files takes maybe a second...very helpful when spins are frequent).

A major drawback to this strategy is that it makes commits to git a challenge (i.e.: the reduced *_DELTA.xls files with few rows cannot be committed directly to the master *_ALL.xls spinner files with 20000+ rows).

Our group is in the process of devising a solution to this major drawback in attempts to further utilize our git. Before we go too far in devising our own custom solution, I wanted to poll this forum to see if anyone else ran into this problem and how they solved it. Any thoughts/ideas related to this unique challenge are greatly appreciated!

James

Hi.

First of all I've no idea why you're using term "spin" and not "commit" provided that you've declared in thread's topic that you're using git repository.

To resolve your issue switch from binary .xls file to a plain text one. This way git will be able to properly calculate diffs thus reducing repository size and increasing commit speed.

If this has nothing to do with git and you just need proper .xls in 3Dx, still consider text files as a raw source of data and use server-side trigger for .xls generation. You can even define external file as data source so that Excel would populate sheets automatically.

Thank you for your response.

The term "spin" may be somewhat more of a local term to my group as I don't see it in the Dassault documentation; I didn't realize that so I apologize, but I wasn't it wasn't intended to refer to a git commit. The 3DX config files are known as "Spinner" (Dassault refers to them as Spinners in their documentation, they are natively *.xls files, which are really tab delimited files and not excel). When they are imported into the schema, we call the following MQL (Matrix Query Language) command (recommended by Dassault in their documentation) that references a Dassault out of the box JPO (Java Process Object):

exec prog SpinnerImport content=schema;

When this command is called, the framework digests only the rows declared; it does not clear out existing rows that haven't been declared in the imported file (additive only). This allows us to create reduced spinner files to declare only the changed/added/deleted rows related to our ticket for significantly reduced spinner processing time. Our group has opted to use the Spinners in this fashion for developer efficiency.

This is not a Git problem in nature; it is more related to our group's practices in how we've chosen to use the files (working with reduced for efficient processing rather than the whole spinners) due to the nature of the 3DX framework. Using the files in this fashion makes publishing the changes to git challenging (more manual in nature without the assistance of a custom tool).

This spinner concept is not new; Dassault has been using the spinner files in this fashion since at least M1 (the predecessor to their 3DX product). Since this issue has existed for a while, I figured our chosen practice may not necessarily be unique to our group and it was possible other groups may be operating in a similar fashion but might have done something to solve the predicament. If there is a simple solution short of devising our own custom tool, I thought posting to this forum would be an appropriate place to find said solution.

I don't believe the spinner files can be changed to binary for improved processing (the command looks for *.xls), but I will look in the documentation for clarification.

In terms of the server side .xls generation, that is a good point and I have thought about that before as Dassault provides this MQL command to export the DBs files:

exec prog SpinnerExport content=schema schematype=* schemaname=*;

However, that command is difficult to use in terms of exporting specific spinner files (i.e.: command lists "Valid schema types are: [program, role, group, person, association, attribute, type, relationship, format, policy, command, inquiry, menu, table, webform, channel, portal, rule, interface, expression, page, dimension, package, index, vault, store, site, location, server]" but I have not found a way of generating the SpinnerTriggerData_ALL.xls file separately on demand). When requesting to generate all with a "*", it will then generate all the spinners including the missing ones like trigger, but also takes a while to export all of the spinner definitions (i.e.: last time I clocked it was 37 minute 54 seconds, which isn't terrible). I'm not ruling that out as part of the final solution, but there should be simpler more efficient options.

I'm sorry if any of that was unclear before; this is my first time posting to the forum and I wasn't sure what level of detail was needed. This question was intended for other users who are familiar with Spinner files and the challenges they face; please let me know if more clarification is needed or if this post should be moved to another section of the forum.

Thanks for further explanation.

General engineering though: have you tried to decompose your giant .xls into many smaller ones?

I'd been developing JPOs and MQL for a couple of years during the first releases of 3DX (2011-2013) and heavily relied on MatrixOne core functionality, but unfortunately had never heard of spinner files. Probably because I never had to push the whole solution to an enterprise level, keeping the whole thing experimental.

It may be a good idea to decompile ootb JPO in question and look for native matrixone API calls it makes. I had been able to fix some bugs this way and it may help you to write a suitable JPO of your own. 

From my experience most of platforms often lack basic maintainability features required for development and customization (test site, incremental dumps, debug and developer modes etc). It may be possible a feature you're talking about may be overseen by DS and now it's up to you to implement it.

Yes, I'm getting the feeling that is the case.

I'm not familiar enough with the architecture to say how easy it would be to decompose the spinners or if it's even possible; that's something I'll have to look into.

I've looked at the decompiled ootb SpinnerImport JPO, but it's pretty barebones. I didn't dive into the native API yet, so that'll be next on my list.

Thanks for the suggestions!