A New Chart Making Program. Update 21 Feb 2018

[SteveM]

I work off of a golden copy of my pattern PDF so I don't need or want to create a new one despite the new features. Whenever you get around to releasing it I'll give it a try, but no rush for me.

[richardandtracy]

Steve, I can completely understand that.




The PDF unit I've written is intended to be a general purpose unit so I can use it in other programs. As a result it's a bit more functional than I need simply for BlendThreads, and has a few more lines of code than absolutely necessary, but is coming in at about 1700 lines of code. The amount added to the main window was a further 2200 lines, and I think I'll have to add a bit more to make better use of it.

At the moment it appears as a single 'Save PDF' button on the 'Define Printout' window under the 'Margins' button. I think it really needs its own 'Define PDF' window so other page sizes can be chosen properly, and then I can also limit the font availability to ones that can be in the current iteration of the PDF unit. Going to have to think how this is going to be used - rather got carried away with the actual realisation it's possible for me to program the creation of pdf files.

Regards,

Richard.

[richardandtracy]

I've taken a big, and potentially worrying, step today.

The programming suite I'm using is copyright 1999. As a result of looking around at the main forum discussing the software, I have realised that that there has been a huge amount of development of it over the intervening years, and a number of functions I've had to create as library items are now available as standard. Furthermore, it is supposed to work with all the legacy stuff from all previous versions. Which is really attractive.
The software has been on sale for approx. £240 for a year or so (since I became aware of the latest version) with versions above the basic being £1200, £3400 and £5600 dependant on functionality. Then, yesterday I saw the basic version had dropped to £48 for a short time. That means my June Pocket money is now spent.. and I've got a bit of learning to do before trying to see if I can get all the old software to work on the new system - assertions of code portability and compatibility are occasionally unfulfilled.

I hope it works..

Regards,

Richard.

[SteveM]

Welcome to the 21st century. I bet there was an asterisk and lots of fine print next to the statement of backwards compatibility. If you can get it working and port it over to using the built in library functions maybe there will be an increase in performance. Computers have changed a lot in the past 17 years so I would imagine that compiler optimisation techniques have changed too.

[richardandtracy]

One of their most useful improvements is likely to be the ability to open PNG files.
I've had a look there me of it, the help file is a .chm file and less well laid out than the old one, there are a myriad of components that can be used, like GPS components, touch screen components, and even screen keyboards. I doubt if I'll ever work out what most of them do.

I hope the compiler works as well as the old one, and if it's better optimised, I'll be happy. My main concern with upgrading is simply getting to know how to use it. Thus far (with 20 mins exploring) I've failed to get any program to compile - I need to find where to specify the compile format (Win32) and not Win64/Android/IOS - despite the fact the bottom of the range one can only do Win32... Seems a redundant flag to me.

Regards

Richard.

[richardandtracy]

I have found that there is now a transparent draw facility in the compiler, so the painful addition of pixel colour components in the 'Backstitch over High Res Image' would take one statement, and not a 50 line series of nested loops. Hrrumph.
I shall have to investigate further before migrating the chart creator over, but it is looking fairly good so far.

Regards,

Richard.

[richardandtracy]

Right.
I have got the program to compile with the new compiler, and after sorting out about 400 errors, it works. The source of the errors was mostly due to the new version of the compiler going over to Unicode characters, but all the text I'd put into warning boxes compiled as standard ascii characters, causing two errors per error warning box. The conversion was surprisingly painless otherwise.

The new compiler has .png compatibility built-in, so can now open .png images for conversion too.

I have done a fair bit of work to the PDF unit to make it more usable, and have now added a 'File|Save PDF' menu option which opens a dialogue box similar to the printing one, but with the basic PDF format standard fonts available instead of the whole TrueType range installed on the PC. Also I've added a little window that allows the PDF document paper size to be specified independently of what the printer can cope with, and have different margins from the printer (as I type this, I realise the pdf file specific data is not saved in the project file & needs to be re-entered each time, will have to add that later). I have built in 27 different ISO and ANSI paper sizes from HUGE down to Post-it size, so hopefully everyone can find a size to suit. The PDF files are large as I've not included any compression (my mind goes to mush when looking at the explanation of what needs to be done, talking about variable 9 to 12 bit byte lengths etc), but they zip up very well. A PDF Creator chart may be 23Mb, zipping to 22Mb, while BlendThreads' PDF files, for the same chart start at 37Mb zipping to 4Mb. The resolution of the symbols may also be altered to reduce size or improve clarity (33Mb up to 51Mb for the example), though the amount added by the highest resolution to the zipped file is so small it's almost not worth bothering about. The symbol resolution is programmed as 300, 600 or 1200 dpi. Give a yell if you need higher (or lower) resolutions.

I cannot notice many speed improvements from the new compiler despite activating a setting for 'Compile for max speed', but there we go. The downloads are available now at the usual links in my signature, and now are 7.6Mb and 7.8Mb for the .zip and installation files respectively. The first compile added 3Mb to the .exe file size simply by going to the new version!

I do hope, I really do hope that it's useful to someone, as there are hours & hours & hours of work gone into this latest update, and it would be nice to know that it's not completely wasted.

As usual, if you have any problems or suggestions, please let me know.

Regards,

Richard.

Edited to add:
Would the program seem less scary for first time users if the window below opened as a 'Wizard' window to guide the user to the right first command?

[SteveM]

Did you make changes to the conversion window? I changed the merge threshold but there was no "update" button visible so my change didn't take effect.

Yep, seems to run about the same speed as before. I've looked at this in the past and notice it isn't for lack of CPU or memory resources so today I opened the resource manager while it was busy applying a change in the conversion window. I noted that the CPU load was about 25%, memory usage was reasonable, and it wasn't trying to access the disk or network. While poking around I found "Analyze Wait Chain" and it reported that BlendThreads.exe was waiting on splwow64.exe which appears to be used to allow 32-bit programs to connect to the print spooler on 64-bit machines.

I got cute and stopped the Windows Print Service but then when I launched BleandThreads it gave me and RPC Server Unavailable error with code 1722 and any attempts to open a chart image were met with memory access errors. Then I tried leaving the print service running till I was in the conversion window and stopped it there, but I saw no speedup and it was still waiting on splwow64.exe. I don't know why the dependency on the print spooler when not actually printing, but this does appear to be causing a lot of sitting around and waiting.

As per the greeting window, it doesn't scare me either way, especially if it can be disabled by a preference setting. It does remind me though that I often struggle to remember the difference between opening a project chart image and a project file so maybe this window could be a little more intuitive as you've got the room to add tips or hints about the function of each button.

Oh, and I was able to open a png image, that was handy and the files are about half the size of a bmp.

[richardandtracy]

In order to work out the shape of the page, most of the windows need to refer to the system printer, by adding the <printers.hpp> header to the list of included files. I imagine this is where the service is being accessed - that sort of knowledge about Windows is way beyond what I normally cope with...
I don't know that this is soluble without a total rewrite. I will look to see if there is anything I can do, but I think not.

I made one change to the thread conversion window, making scroll bars visible if the window was too big for the frame. The new version has them off rather than on by default. I will look at the merge threshold update, see if I forgot to make it auto update before.

I shall incorporate the front page and make it optional too.

Regards

Richard

[richardandtracy]

I have done a bit of work on the program in the last week, and have made some progress.

Bug fixes:
In the Thread Conversion window as a result of SteveM mentioning it, I did find an oddity with the Merge Threads Up/Down control. For some reason it wasn't triggering the 'OnClick' event when it was clicked. I have had to put in an 'OnMouseUp' event instead to get it to trigger.

I have detached the Up/Down controls from their adjacent edit boxes and put a separate variable behind the two which is altered when either the edit or the Up/Down is altered. The built-in behaviour did not always respond to a change in the edit control, even though it was supposed to according to the Borland documentation from 1999, and this erratic behaviour is still apparent in the Embarcadero compiler I've now got, 6 generations on.

Other little improvements I've managed since the last update, and will be incorporated into the next upload:

1. The Mock-up can now be printed as the last sheet of the chart. The title line gives the scale as 1:1 or 1:[scale] if it has been shrunk to fit the page.
2. The Mock-up can be added to the PDF file, as the last sheet. Furthermore, you can specify that the mock-up sheet only is created as close to 1:1 scale as the page range will permit. So, if you are using the ISO A series of pages, it can automatically select the most appropriate page size from A0 to A6, similarly with the Ansi Letter range (Letter/Ansi B/Ansi C.. Ansi E), ISO B series (B0 to B6) and the Ansi Architectural sheet size (ArchA-ArchE). If the largest page in the range is not big enough, the biggest page size is used and the mock-up is shrunk to fit that.
3. I have done a tiny tweak to the PDF generator that took hours & hours to get to work (it needed a different encoding line in the font descriptor object - really not clear what I needed to do in the PDF 1.3 Specification document, I had to correlate a table on page 296 with a paragraph on page 74 and a section at pages 132/3, yeah, it was that clear), and the PDF output now shows copyright & similar symbols correctly.
4. I have added keywords to the PDF file, so "Cross" "Stitch" "Chart" and "Pattern" now appear as PDF keywords. The Author field of the PDF file is made the same as the Chart Author line on the chart front page.
5. I have added the little 'What would you like to do' window shown above, but also added a 'Paste Chart Image' option if the clipboard contents are suitable and made the window respond to defaults so you don't have to have it open each time if you don't want it. I must admit, I have found it surprisingly useful while testing the program for speeding things up and reducing mouse clicks - so don't set it off by default until you've tried it a few times.

Things I do not think are likely to happen in the near future:

1. Embedding fonts in the PDF file. I have been unable to find anyone who could help me on this, and I have no idea how to go about it on my own. The PDF Spec document I'm referring to has been written buy a hand-wavy type person, rather than someone who needs to use the document, as a result, useful information is hard to glean from it. Then add to the mix the fact that the compiler libraries I'm using has been written expressly to hide the low level stuff (so the programmer doesn't have to waste time with it if they don't need to) and I'm slightly stymied from both ends.
2. Speeding up the thread conversion by eliminating the effect of splwow64.exe. I have asked on various forums about this, and no-one willing to answer has suggested solution methods I could try. I have tried adding a cpu thread that opens the 'Thread Conversion' window, but there are too many variables I need to pass into and out of the 'Thread Conversion' window to be able to co-ordinate these with a cpu thread. The current help file documentation is very sparse, saying 'Embarcadero Technologies does not currently have any additional information' after a 2 line descriptor of the TThread object I'd have to use. Unfortunately the 17yo BCB4 documentation does not cover multi-thread programs well, so relying on that instead and applying it to the new version would be difficult too.

Hope this 'State of Play' is of interest.

Regards,

Richard


Edited to add:
I have had a stab at SteveM's idea of preferentially selecting solid colours when converting blended images. I think I may have been able to incorporate a solution that works with negligible performance impact. I have incorporated 4 levels (None, low preference, medium preference and high preference for solid colours). On the image I have tried it on, the high preference gives about 40% blends, while the no preference option gives about 90% blends.

[SteveM]

I don't mind hearing about the gritty details, though I have very little experience developing with an IDE and extensive set of propriety libraries, or an OS that fires off blocking processes like that. Nearly all of my C coding experience is with gcc and most of that for embedded systems such as the Atmel AVR and ARM7 with either no OS or an RTOS. But for the most part I follow you.

I am very excited to hear about the weighting towards solid colours and can't wait to give it a try.

[richardandtracy]

Have updated the program with the changes listed above, and one other:

Now, when you click on the 'Chart Page Cross Reference' image, a colour preview of the chart pops up, opening at the page you clicked on. This is, I admit, brochure engineering. Utterly pointless, but made me happy.

Regards,

Richard

[SteveM]

Well I am a fan of this update!

I'll get my only gripe out of the way - the new colour preview uses a page size of something like 80x100 instead of my selected size of 60x80. This simply means the pages don't line up, all of the pattern is there to be previewed which was kind of nice actually as one can get a better idea of the confetti level of various areas.

As much as I have enjoyed the new feature of allowing a preference for solid colours over blends, I really enjoyed the ability to uncheck "Update on Setting Change" and having all controls work exactly the same in each mode. I find it a huge speed up to be able to punch in the settings that I need to replicate my "Ionian Mission" pattern then click on the update button. It used to take ages to click my way one update at a time to the required settings.

I am very happy about the ability to prefer solid colours over blends and used my "Ionian Mission" pattern as a reference to compare the three levels available to my original. I took screen shots and compared them and on the "low" setting there was barely any difference and even on the "strong" setting there were only one or two spots that degraded a bit, but certainly acceptable to many.

Here is a comparison, top left is the "no preference", top right "low", bottom left "medium", and bottom right "strong" preference for solid colours. What I notice is that the solid colours increase the error so the dither starts to look a bit more aggressive and in one or two small areas starts to become noticeable.



Here is what I got in table format:

Code: Select all

               | Original/Off | Low | Medium | Strong |
=======================================================
Thread colours |  356         | 355 | 352    | 341
Symbols        |  351         | 358 | 357    | 354
Solids         |  16          | 51  | 74     | 98
Blends         |  335         | 307 | 283    | 256

The number of symbols used is fairly constant which is what we want, the shopping list of thread colours to buy goes down ever so slightly, but of course the desired effect is that we do see a range from about 5% solid colours to a max of 28% solid colours for this pattern. All in all a very good tool for those willing to accept a slight amount of error to have fewer blends.

[richardandtracy]

Glad it's useful for you Steve.

Size: possibly the symbols per inch is different, maybe the margins, or possibly the page size of the selected printer is not the same. The behaviour is not standard Windows behaviour in that the program deliberately searches for the last printer with 'PDF' in the name, and only if it can't find one does it default to the Windows printer - I did this to stop wasting paper when testing the program. It may be worth saving the printer name, page size and symbol numbers in the project file to ensure everything is exactly repeatable between sessions. I shall have to see if the actual printer page size name is visible to the program. I think it's hidden, but there may be other ways of getting to the name.

Have to confess I can barely see a difference between the four pictures you show. I am looking with a tablet & the screen is small, but it is difficult to spot any change. I was going to ask if you thought the levels are right, but get the feeling from your results and my testing too that it's about right. Any stronger preference for solid colours and you shouldn't be blending...

Regards,

Richard.

[SteveM]

The differences really are quite minor which speaks to how well you've implemented it. You pretty much have to zoom 200% or more to start to see the detail of what looks different. For example, on second ship's sail near where the foreground ship's sail meets with it you can start to see a vertical line of light blue stitches on the "strong" setting. But you won't notice it just putting your nose to the glass at 1 pixel = 1 stitch.

There are but two other areas where I can spot a difference between no preference and strong. There is a dark area near the bottom of the sail that begins to look grainy instead of more smoothly blended, and the dark water just in from of the bow wave begins to get a green dither that could be a bit distracting. I have no issues with the "low" setting and could probably be happy with the "medium" setting for this pattern. Tomorrow I'll test some other images to see if the reduction in blends seems appropriate, but I do agree that if you're starting to lose the blending effect then you may as well be using all solids.

[richardandtracy]

The technical bit behind how it's done:
For each pixel, the program checks to see if it's the same as any previous pixel. If yes, it uses the selected replacement colour. If no, it runs through every possible thread, calculating the square of the red, green & blue errors for each thread and saving this in an error array. After doing this, the program enters another loop to go through the 11000+ long error value array looking for the lowest errors. It stops at the end of the solid colours, which are always the first 400+, and saves the error value and array reference pointer. Then the lowest blend error is found, keeping the error value and pointer.
After finding the lowest blend error, the solid colour array pointer is substituted if it is lower when the blend error is multiplied by the 'preference factor'.

The preference factor is 1 for no preference, 2 for low, 3 for medium and 4 for strong preference.
I did wonder initially if it shouldn't be 1, 2, 4 & 8 instead, but decided to try 1-4, which seems to work.

Sounds difficult, but it actually only added four variable assignments and one if() statement, which is negligible compared to the 11000+ assignments and 11000+ comparisons it was already doing for each new coloured pixel.

Do you think it worth while to have a way of the user specifying the number of symbols in a chart if less than the Max the page can take?

Regards,

Richard.

As an aside, the new IDE has a code checker to check for code complexity, and comes up with a toxicity quotient for each method.If the quotient exceeds 1 you should simplify the method. There were about 500 below 1, and 100 or so methods above one. The worst was 50.7... Shows I have taught myself bad habits when learning to program.

[SteveM]

This morning I got to thinking that a preference for solid colours over a blend is probably going to be most useful with more "normal" patterns instead of my 351 symbol beast. So I dropped it down to ~150 symbols and found that on the Ionian Mission the percent of solids went from as low as 9.5% to 40% and while I could see the odd difference here and there, both patterns really looked equal to me.

Next I loaded up a piece of art that I've found tends to do with with just solid colours and also set it to about 150 symbols and the range was 13% to 46% solid symbols. Again, either pattern would have been acceptable, but I did see one or two small areas of the "strong" preference pattern that I would want to manually tweak a bit to reduce the harshness of the dither.

Do you think it worth while to have a way of the user specifying the number of symbols in a chart if less than the Max the page can take?

Not quite sure what you mean here, but if related to the algorithm to allow preference for solids over blends I'm happy with it as-is.

[richardandtracy]

Just meant if the page can take 86x107 symbols, you could explicitly specify 60x80. That's all.

Regards,

Richard.

[SteveM]

Just meant if the page can take 86x107 symbols, you could explicitly specify 60x80. That's all.

Ah. Yes, that could be tucked away in a setting somewhere. But I wonder if you can't just pull the info from the "Cross Stitch Chart Information" area of the main window. Right there is says "Each page is 60W x 80H Symbols" so the correct number of symbols per page is known.

[richardandtracy]

I have updated the program a bit in the last month.
Most of the update effort has centred on making the PDF export more robust. I have experimented until I'm boss-eyed without success on embedding fonts, but the rest of it seems generally more robust.

The biggest visible part of the update is the addition of a binary file export (*.bsb - 'inkysloth' may get the reference to Noodler's Bay State Blue fountain pen ink, a vibrant blue but is tricky stuff to live with). This keeps all the information relating to a conversion together in one file. The converted chart image, high resolution image and one backup image are stored along with the thread definition files, symbol cross reference list , colour palette used and all other internal variables & array. The binary file can be quite large (10-12 Mb) but it is blisteringly fast in comparison to opening a text project file.
I tried it with a 7000 colour conversion of a Leonid Afremov painting. The text (.bs) file opened in 13 minutes 49 seconds. The binary file (.bsb) opened in 11 seconds. That's 75x faster. Now, this is an extreme example, with simpler projects getting less benefit. I have not yet found a project that loads more slowly using the binary file, though.

I have kept the text project file capability too.
As yet I don't suggest that you only save the binary file - it hasn't been around long enough to be 100% certain that everything else can be ditched. I recommend that you backup the project using the text project file and use the binary project file as the working file.

There are things I want to add to the binary file (like keeping all the vector graphics used in the image editor, the base image used in the image editor and saving the optional background fabric image visible in the mock-up) but I've set up the format so it will be backwards compatible with all previous versions and no project will be impossible to read.

I have added Kreinik Silk Mori as an optional thread definition file to be added if required.

Again there are two downloads.
The 7.2Mb installation file creates short cuts in the start menu and also adds some registry settings to register 'Notepad' as the viewing app for the .bs and .bsb file extensions. This means you will see the project title shown on the Chart front page in the preview pane of the 'Open Project File' Dialogue box.
The 7.9Mb .zip file doesn't do all the nice add-ons that a full installation does, but has the advantage that it can usually get through blocking anti-virus software.

Hope it's useful to someone.

Regards,

Richard.