• Index
  •  » Bug reports
  •  » Issue with "BackendXML: don't save empty fields" and yesno filters

#1 2018-02-04 04:40:23

van der Decken
Member
Registered: 2018-01-30
Posts: 27

Issue with "BackendXML: don't save empty fields" and yesno filters

The change made to GCBackend/GCBackendXmlCommon.pm: "BackendXML : don't save empty fields" creates a problem with yesno type fields and filters.

A yesno field ought to have a value of 0 or 1. The change at line 268 in GCBackend/GCBackendXmlCommon.pm:

Code:

print $tmpFd '  ', $field, '="', $data, "\"\n" if $data;

will not save a yesno field with a value of 0. When read back in, it appears that the code doesn't initialise it to 0 if it's not present - it leaves it as an empty string. And that creates a problem in the filters, which don't see an empty string as being equal to 0. If you then create a filter on that yesno field looking for a 0 ('no') value, it won't find anything.

I've just found this, and off the top of my head I've come up with a few potential solutions.

1. Revert the change and write all fields out, even if empty.

2. Modify the test to specifically look for an empty string:

Code:

print $tmpFd '  ', $field, '="', $data, "\"\n" if $data ne '';

As I say, I've *just* found this issue, so I'm not 100% sure that this fix would be foolproof.

3. Find the code that does the reading and have it initialize yesno variables that aren't present to 0 rather than an empty string.

4. Modify the yesno filter test to treat an empty string as a 0.

#1 seems foolproof: put things back the way they were. #2, if it indeed works the way I think it should, would give the advantages of not writing empty field tags while restoring the yesno filter functionality.

Thoughts?

Offline

 

#2 2018-02-04 17:30:49

van der Decken
Member
Registered: 2018-01-30
Posts: 27

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

Further notes upon reflection:

Although option #1 will restore the empty fields to the gcs file once you've made a change to your collection and saved it, you still have to edit the gcs file afterwards and replace instances of yesnofield="" with yesnofield="0" (where yesnofield is replaced with the actual field name(s) of your yesno field).

Option #2 requires even more fiddling. First, you have go through option #1 to get the gcs file back to a state where all your yesnofields are populated properly in the file. Only then you can add the modified test to the code.

I had a feeling it wasn't as simple as it appeared late last night...

Offline

 

#3 2018-02-07 16:09:47

kerenoc
Member
Registered: 2016-03-19
Posts: 205
Website

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

I made this modification to reduce the size of collections but I didn't notice any problem at that time, not using filters intensively.


After digging into this, I found that there was no problem with filtering on the boolean field "seen" for films but problems with filtering on "read" for book. The difference is that for films the "notnull" attribute of the field is set to "true". If one set it for the "seen" attribute in GCbooks.gcm, it works as expected.

To fix the behaviour for the "favourite" field, one has to set "notnull" in GCModels.pm.

I added a "notnull" attribute for all field of type "yesno" in models (committed to GitLab). I hope it doesn't break something else!

Thanks for discovering this and helping clarify the issue.

Last edited by kerenoc (2018-02-07 16:10:36)

Offline

 

#4 2018-02-07 21:14:37

van der Decken
Member
Registered: 2018-01-30
Posts: 27

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

Thanks again kerenoc. Modifying the models seems like a much better solution. I liked the fact that skipping writing the empty fields kept the collection file size down (and made it easier to skim through when I was looking at it behind the scenes). I look forward to trying out your change.

Offline

 

#5 2018-02-08 00:07:20

van der Decken
Member
Registered: 2018-01-30
Posts: 27

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

I tried out the notnull change and... it didn't work for me. So I dug into things a bit and discovered what was going on.

It turned out to be the way that the init tag was set up.

The particular field I was using in the test (the cancellation yesno in GCstamps if anyone's wondering) had init="".

Which I took  to mean "if not specified in the item in the collection file, this field in the item is initialised to an empty string".

I changed that to init="0" and the filter started working as I expected it. Items where the cancellation field wasn't set to "1" in the collection file were now defaulting to "0" rather than "".

Out of curiosity, I tried setting init="1" on the cancellation yesno field to see what would happen. As I expected, everything was now showing up as a "1".

Which raises another issue... there are yesno fields in GCboardgames.gcm, GCgames.gcm and GCsoftware.gcm where init="1". If this indeed works the way it appears to, there is no way to set those fields to "0" and get it to stick. A "0" value won't be written to the collection file when it's saved, and when it's read back in, it'll default to "1". And the next time it's saved, that field gets set to "1" for every item in the collection file. Oops...

Last edited by van der Decken (2018-02-08 00:27:16)

Offline

 

#6 2018-02-08 13:44:32

kerenoc
Member
Registered: 2016-03-19
Posts: 205
Website

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

My current view would be to restrict the use of the "init" field as the default value when a new collection item is created.

When a collection is loaded, for fields with the "notnull" attribute, one could set the value to 0 if the field is numeric, to an empty string otherwise. I'll check that this evening.

Offline

 

#7 2018-02-08 20:00:20

van der Decken
Member
Registered: 2018-01-30
Posts: 27

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

kerenoc wrote:

My current view would be to restrict the use of the "init" field as the default value when a new collection item is created.

I was going to say that that function seems to be already be mostly fulfilled by the settings in GC<modelname>.default.gcs file that's created in Settings -> Default values for new <item name> (but you have to actually choose to create that file)... but as I looked into things briefly, it got murky.

Yes, the init field does appear to work the way you stated above... for some other field types. For example, there's a couple of short text fields in the GCstamps model that have init values defined, and those init values don't magically re-appear when I empty out the fields. So why do the inits for the yesno fields not work that way too? Curiouser and curiouser.

Update: Actually, strike that last paragraph. Init values for other field types are popping back up in the default file the next time I load the program and emptying a field too. I'm guessing that my default file was made before I put the "don't write empty fields" patch in.

Last edited by van der Decken (2018-02-08 20:46:15)

Offline

 

#8 2018-02-09 07:25:33

van der Decken
Member
Registered: 2018-01-30
Posts: 27

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

Here's a one line patch to the setFilter function in GCDisplay that adds the same 'nulls are really 0s' preprocess to yesno type fields that's already done for number type fields. That seems to fix the original problem I saw (it's in fact #4 in my original list of possible fixes). With this, it's not necessary to write 0 value yesno's to the collection file, or set init="0" for yesno fields. I will keep an eye on this here to see how it behaves but so far so good.

This doesn't address the init values being set for missing yesno fields on collection file read, or init values being written out to the default file when they're present in the gcm file but then erased by the user in Settings -> Default values for new <item>, but one down, two to go.


Attachments:
Attachment Icon yesno-filter-nulls-ok.patch, Size: 713 bytes, Downloads: 55

Offline

 

#9 2018-02-09 23:35:46

van der Decken
Member
Registered: 2018-01-30
Posts: 27

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

I think I've pretty much got it figured out now.

Not storing false field values in the collection file was causing yeno filters to fail when testing for a yesno value  equal to No (i.e. 0). The filter test uses the string equality test for "=" test for all field types. As it turns out , the string equality test 0 eq '' is false in Perl. The numeric equality 0 == '' would have been true and so would have worked. It'd been ugly though.

Adding the 'check for null numbers and replace them with 0' test to the code in GCDisplay as in the previous message does get around this. A more 'proper' solution could probably involve making sure that yesno types set $numeric = 1 and passing $numeric to the compareExact function and using the numeric == for numeric types and the string eq for strings.

That takes care of the yesno filters failing on a No test when false values weren't store in the collection file. But there's still the question of why init values were being used when reading a false value from the collection file, but only on the yesno types, and why they were cropping up in the default file after being erased.

First, the answer to the yesno types being populated with init values when the collection file was read in. I'd added the notnull="true" tag to the gcm model file as suggested by kerenoc, found it didn't solve my problem with the yesno filters, and then added in init="0" as well on my yesno fields. Here's what happened when a collection file was read after that:

1. A field is defined in the gcm file with notnull="true".
2. When a collection is read, as each item is parsed, the fields defined with  notnull="true" are examined to see if their values are false (anything that stringifies to the empty string or the string '0' is false.). False values are replaced with the init value from the gcm file.

So what notnull="true" is doing is "if I read a false value from the collection file with this value set, replace it with the init value. That explains why notnull="true" and init="0" TOGETHER also appeared to solve the yesno filter problem. Empty fields were being replaced with "0", which passed the eq test in compareExact.

Once I'd added the 'check for null numbers and replace them with 0' test to the code in GCDisplay, I could take out the notnull="true" and init="0" entries from the gcm file and put it back the way it was.

That also explains why other fields that did have init values defined weren't replacing item field values of empty strings or 0 numeric values; none of them had notnull="true" defined as well.

All that's left to explain is why the init values from the gcm file aren't being overridden by the values I set up in Settings -> Default values for new <item>. It all comes down to not writing out empty strings. There are a few fields in the GCstamps model with init values like the string "0 euros". I changed them to empty strings because I didn't need those particular fields populated.

That worked fine when all strings, empty or not, were written out. The model file would be read in, and any field init values stored. Then the defaults file would be read, and my settings would overwrite the model defaults. But once empty strings were no longer stored in the defaults file, there was nothing to overwrite the model defaults, and the init values reappeared.

My first thought is that the save routine in GCBackendXmlCommon might need to be called with an extra parameter when saving the defaults file that tells the code to write out empty field values. That would allow us to skip writing empty fields to the collection file (saving time and space) but still writing them to the defaults file, where they're definitely needed to erase a non empty model init value. But as I've literally just now figured out this last bit, there may be a better way after some reflection.

Offline

 

#10 2018-02-10 02:33:43

van der Decken
Member
Registered: 2018-01-30
Posts: 27

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

I made a minor change to the code that saves the collection & defaults files to ensure that all fields are written to the defaults file, while still only writing non empty data to the collection. So far it appears to do what I want it to, but I'm going to be keeping an eye on it for unintended side effects. A patch file us attached for anyone who wants to try it out.


Attachments:
Attachment Icon write-all-fields-to-defaults-file.patch, Size: 1,529 bytes, Downloads: 70

Offline

 

#11 2018-02-10 11:30:10

kerenoc
Member
Registered: 2016-03-19
Posts: 205
Website

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

Hello,

last evening, I integrated your previous patches and a modified GCBackendXMLParser into the Test branch on Gitlab. Unfortunately, when I try to download your newest patch file from the forum I only get an empty file and can't check if it's consistent with the changes I made.

Offline

 

#12 2018-02-10 17:26:01

van der Decken
Member
Registered: 2018-01-30
Posts: 27

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

That is odd! I'm getting much the same thing. My browser says it's downloaded but when I check the download directory... nothing. I will recreate the patch from scratch and upload it again.

Offline

 

#13 2018-02-10 17:54:18

van der Decken
Member
Registered: 2018-01-30
Posts: 27

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

Okay, let's try the patch file one more time.

Afterwards: I've just downloaded it and it was there this time.

Last edited by van der Decken (2018-02-10 17:55:19)


Attachments:
Attachment Icon write-all-fields-to-defaults-file.patch, Size: 1,572 bytes, Downloads: 55

Offline

 

#14 2018-02-10 19:06:17

kerenoc
Member
Registered: 2016-03-19
Posts: 205
Website

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

I can now access this patch file too.

If saving and oading files without all fields is working, we may not need this patch as the collections' size will then be optimized.

Offline

 

#15 2018-02-10 19:13:23

van der Decken
Member
Registered: 2018-01-30
Posts: 27

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

The 'write-all-fields-to-defaults-file.patch' file only affects the GC<model>.defaults.gcs file.

It write all fields, empty or not, to the defaults file. This allows the user to override a non empty init value from the GC<model>.gcm file with an empty field, that's all.

Empty fields will still be skipped when writing to the collection file. It's done from a different call, which doesn't specify the extra parameter.

Offline

 

#16 2018-02-10 19:35:11

kerenoc
Member
Registered: 2016-03-19
Posts: 205
Website

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

Ok, I understand. I had never used these configuration files before.

Thanks for the contribution that is now on Gitlab.

Offline

 

#17 2018-02-10 19:52:24

van der Decken
Member
Registered: 2018-01-30
Posts: 27

Re: Issue with "BackendXML: don't save empty fields" and yesno filters

I'm glad I was able to clear things up. Thanks for adding it to Gitlab.

I'm still at the stage with GCstar of "I wonder what this does... oh, that's what it does" which is sometimes followed by "that's useful, I'm going to use that" or "huh? Why does it do that?" So I'm still exploring the hidden corners of the UI, clicking buttons and searching through menus to see what there is to see. I've already stumbled across a few more "huh? Why does it do that?" items that I'm in the process of trapping in a corner.

And I hope I'm not coming across like I'm complaining - far from it, I think GCstar is GREAT. Had I stumbled across it years back when I put together my Meccano inventory database, I probably would have used it for that purpose too rather than roll my own.

Offline

 
  • Index
  •  » Bug reports
  •  » Issue with "BackendXML: don't save empty fields" and yesno filters



Should you have a problem using GCstar, you can open a bug report or request some support on GCstar forums.