kernology - where laurie and learning meet
  • Home
  • Blog
  • Pictures
    • Art & Craft >
      • Temari
      • Yubinuki
      • Quilting
      • Weaving
      • Lapidary Work
      • Metal Work 2010
      • Metal Work 2009
      • Classes - 2010
      • Classes - 2009
    • Boots
    • Views of Texas
    • Views of Scotland
    • Travel >
      • Bunny Bravehart Road Trips
      • Scotland 2012
      • Maine 2011
      • New Mexico 2010
      • Scotland 2008
  • Oh, the places you will go!
  • Contact Laurie

Byte Me!

12/15/2013

 
I have been on a new project at work for about 3 months.
I am writing an application to read a data file and then transform it to another format but before we accomplish that, I have to learn the data format and what it all means.

For the past month and a half, I have been reading the specification and writing a program that will read the data, grab the header information, and put the result into an XML file.  That part is now done.

All the data to be read in from the original file is in binary format and that is what I have been dealing with. I have not worked with binary data in over 35 years. I know how it is worked with but everything I learned back in college is still valid in theory but the actual tools to use have changed and that is what I am working to learn.

Let's review shall we - 
Binary format is data that is either 0 or 1. This is also known as a base 2 numbering system.
[FYI, the numbering system we use in our every day life is base 10]
A BYTE is composed of 8 bits
00000001 = 1
00000010 = 2
00000011 = 3
etc.
There is such as thing as a nibble which is 4 Bits
And there are 2 nibbles in a Byte

A "word" is n number of bytes, but n depends on the type of computer system you are working on.
So way back when we got our first PC's that were 16-bit, that implied there were 2 Bytes in a word; in a 32 bit system, there were 4 Bytes in a word; and of course now that most of us are on 64 bit systems, we have 8 bytes in a word.

And when dealing with binary data; YES, you can Byte off more than you can chew (in a nibble) - ha ha.

Well my data file is made up of packets of 2 Byte words so before I can even start to read the data, I have to first swap the byte positions such that the byte in position 0 goes to position 1 and  what was in position 1 goes to position zero this is because the system writing the data is slightly different than the system that reads it on the computer I use.

After that I have to look for 3 Bytes (to be known from now on as the magical series). 
The values which are: 0xFA, 0F3, and 0x20. 
That is hexidecimal, which is base 16 which is counted as 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A (in lieu of 10), B, C, D, E, and F (15 in decimal). So in normal speak I am looking for 250, 243, and 32 decimal values.

BUT get this, I can't just search for them across all the bytes; that would be way to easy. It turns out that the system that writes this data uses an odd system of bytes that are 8-9-9-8-9-9-8 bits but with a 8 bit writing to the file. SO this magical series of data that I have to look for is split across normal 8 bit Bytes but is contained in the weird series. 

How does one find those values.... well that is what bit masking for and that alone is a whole blog post on it's own. Lets just say, it is like subtraction to find how many 2 you can actually see in the number 12 or 2,523 and what positions they are in. Bit Masking allows you to see if the 2 is in the first (12) or last position (2,523). 
 
This image shows how I determined what the bit mask should be.
Picture
Once I find a magical series, I have to break up that section of the packet of data into 2,000 byte lists and then find the next magical series. 

But wait, there is one more twist. Occasionally the system that writes the data can toss out an extra bit or 2, or 3, and if this happens, all the rest of the bits are shoved off their position, I have to shift all the data x number of times (0-7 for a total of 8) and search for the magical series, again. Once the series is found, I can pull that data out and start searching again.
 
After 10 days of working on this bit of code (ha, ha a pun!) I have been able to process one channel (of 6) in one file.  It is a big accomplishment as I can now move on to the next part which is to read a clock and assign a running time value to each piece of data. 

Here are the numbers of what I was working in case you wanted to know:
  • 9,378 packets of data in the file.
  • With an average of 8,182 bytes per packet
  • Total: 76,730,796 bytes to be searched or 613,846,368 bits.  

Comments are closed.

    RSS Feed

    About Laurie

    Laurie lives in central Texas with Erich, a.k.a. "the shop elf", her hubby of 35+ years and Cowboy Boots, the cat; her metals studio including 100+ hammers and 300+ chasing tools; her sewing studio which has a sewing machine, a closet filled with fabric, hundreds of skeins of embroidery floss and perle cotton, silk and other materials, and Mrs. King the dress dummy; two weaving looms, assorted knitting needles, tubs of yarn; lots of books; plus a plethora of geeky tech gadgets, computers, and more.

    Archives

    March 2023
    February 2023
    January 2023
    December 2022
    November 2022
    October 2022
    September 2022
    August 2022
    July 2022
    June 2022
    May 2022
    April 2022
    March 2022
    February 2022
    January 2022
    December 2021
    November 2021
    October 2021
    September 2021
    August 2021
    June 2021
    May 2021
    April 2021
    March 2021
    February 2021
    January 2021
    December 2020
    November 2020
    October 2020
    September 2020
    August 2020
    July 2020
    June 2020
    May 2020
    April 2020
    March 2020
    February 2020
    January 2020
    December 2019
    November 2019
    September 2019
    June 2019
    May 2019
    April 2019
    March 2019
    February 2019
    November 2018
    October 2018
    September 2018
    August 2018
    July 2018
    June 2018
    May 2018
    April 2018
    March 2018
    November 2017
    October 2017
    September 2017
    August 2017
    June 2017
    May 2017
    April 2017
    March 2017
    February 2017
    October 2016
    September 2016
    August 2016
    June 2016
    May 2016
    April 2016
    February 2016
    January 2016
    November 2015
    October 2015
    September 2015
    August 2015
    July 2015
    June 2015
    May 2015
    April 2015
    March 2015
    February 2015
    January 2015
    December 2014
    October 2014
    September 2014
    August 2014
    July 2014
    June 2014
    May 2014
    April 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    September 2012
    August 2012
    July 2012
    June 2012
    May 2012
    April 2012
    March 2012
    February 2012
    January 2012
    December 2011
    November 2011
    October 2011
    September 2011
    August 2011
    July 2011
    June 2011
    May 2011
    April 2011
    March 2011
    January 2011
    December 2010
    November 2010
    August 2010
    July 2010
    June 2010
    May 2010
    April 2010
    March 2010
    February 2010
    January 2010
    December 2009
    August 2009

    Categories

    All
    42
    Android
    Boots
    Bunny Bravehart
    Codesmith
    Comic-Con
    Covid
    Day Job
    Family
    Fencing
    Fiber
    Health
    Hike
    Jewelry
    Knitting
    Laurie
    Metals
    Minecraft
    Pandemic
    Photography
    Quilting
    Retirement
    Road Trip
    Scotland
    Sources
    Spinning
    Temari
    Texas
    The Big Walk
    Weaving
    Web
    Yubinuki

Powered by Create your own unique website with customizable templates.