Dangerous Prototypes

Seems like this question comes up a lot, but I've never seen a good solution for this. I currently use Google Spreadsheets to keep track of all my parts, but searching through it sucks. It is difficult to know what parts I need to restock when I start a new project. Usually takes me an hour or more to go through my BOM and spreadsheet to find out what I need.

So, I create a utility to do this for me, which has turned into a web app using the Octopart (http://http://octopart.com/) API. The idea is you upload your BOM from Eagle and the app tries to find the cheapest distributor for the parts you don't have. It also keeps an inventory of your existing parts.

It looks like this. Would anyone find this useful?

Looks nice. I'd use it if it was not Eagle-centric (I use Altium Designer).

Actually the site takes a CSV file so it could potentially support other CAD tools, but that is something I will defiantly try and make easier for non Eagle users.

Since I am using the Octopart search engine API the site will support the ~12 or so distributors they work with. That includes the major ones like Digikey, Mouser, Newark, Future, etc... Did a google search, but couldn't find a complete list.

And it would be nice to have an option to eliminate certain distributors from the search as well, as some of them do not ship to certain destinations or have other issues which may preclude some people from choosing them.

Good point. Right now it is sorted by number of parts found then price, but it should be easy enough to add a preferred list of distributors. Thanks for the feedback.

Hi

This is a great idea, I am interested in it as it stands.

This project from a German hackerspace is also developing well http://partkeepr.org, but it is more of an inventory only system.

I am also interested in your idea from a different angle, as one of my many projects is to do something similar with all the DP project BOM to extend the current BOMs and Shopping carts http://http://dangerousprototypes.com/docs/Shopping_carts.

Ian published one view of my idea above, but it needs something like Octopart to keep at least the prices and availability up to date.

Implementing this effectively allows users to purchase any project as a "kit" of all the major components from their preferred supplier(s). Exclusions would be mainly be the PCB and of course out of stock items.

The guys at Upverter, which people here love and hate, are already doing something similar where they presumably use Octopart to purchase the parts for Upverter projects for you.

Also have you looked into including delivery costs?

Are you interested in extending in this direction or releasing your code, so others can?

Spanner888

This project from a German hackerspace is also developing well http://partkeepr.org, but it is more of an inventory only system.

Partkeepr looks cool. Seems like a robust inventory system.

I am also interested in your idea from a different angle, as one of my many projects is to do something similar with all the DP project BOM to extend the current BOMs and Shopping carts http://dangerousprototypes.com/docs/Shopping_carts (http://dangerousprototypes.com/docs/Shopping_carts).

This is a great idea. One of my goals was to have API that would let people embed their parts and projects on their own sites, but I never thought about using it in a shopping cart. Seems like the perfect use case. Currently, the website generates a CSV that the user would copy and paste into the various BOM importers like this one https://ordering.digikey.com/registered ... eader=link (https://ordering.digikey.com/registereduser/bomwizard.aspx?wt.z_header=link)

The guys at Upverter, which people here love and hate, are already doing something similar where they presumably use Octopart to purchase the parts for Upverter projects for you.

Haha, I like these guys plus they're based in Toronto! I figured they would add some kind of BOM manager eventually, but I think there would still be people that want a stand alone product that does inventory, ordering and collaboration. Seems like you agree.

Also have you looked into including delivery costs?

On the backlog. Not sure how I would do it. Maybe just hard code the values for each distributor?

Are you interested in extending in this direction or releasing your code, so others can?

I'm unsure if I will release all of the source. I am planning to releasing the parts the would be useful to the community like the octopart scripts, related tools and inventory system, but probably not the collaboration features. It depends on how/if I plan to monetize it.

I am writing a program that directly searches: Mouser, Digikey and Arrow, tells me the build cost of a board (and withdrawals the needed inventory), makes CSVs to cut and paste to make orders at places, imports order invoices to keep part counts up to date (from PDFs Digikey/Mouser and HTML: arrow).

If parts are needed it calculates the lowest cost order to meet a build need list trying to account for shipping. It also looks at part price breaks to try to reduce running costs (IE it will not order 49 of something if 50 will cost less, and if you give it the percentage overhead it might order 50 of something you only need 40 of today; If you can source the part list from two places or a single place is the single place is less than the shipping overhead it goes to the single place, if the two (three, N) places are less it uses multiple sources).

The data is stored in CSV, and the program is in Java. My code will be MIT licensed when a functional version is complete. Its primary goal is minimization of running costs of building small batches of known designs.

Wow! That's amazing brian! Any reason you didn't use a part search engine like Octopart or Findchips? Also, how difficult has it been parsing the distributors websites? I find my self doing this for different projects and I'm always worried their website will change.

I would really like to see an open source, online inventory and project BOM manager. I have tried the german hackerspace package, and written a few scripts of my own.

We have had discussions with the octopart people about a BOM uploader, or using their API for preparing BOMs for our projects. Not much has come from it though. One issue is its hard to get fuzzy descriptions to give good part suggestions. We tend to write descriptive BOMs because parts are interchangeable: "1uF 16v+ 0805", not manufacturer part numbers like "MDF-517-63409-1016A". Complete part numbers on parts like LCRs, in most cases, increases the price of our stuff because Seeed presumes they have to import that specific part.

Internally, we have a zygote of an ERP system that tracks the inventory of parts we have, but does not have any ability to remove parts for projects or work with eg Eagle BOMs. I would like to release our ERP and management system under MIT license, but at this point a ton of internal data, passwords, and a few things we don't share about DP are hard-coded into (not to mention the mess of php).

Internally, we have a zygote of an ERP system that tracks the inventory of parts we have, but does not have any ability to remove parts for projects or work with eg Eagle BOMs.

Is this custom built and is this what most companies do? I would like to know how small to medium size business deal with this. I'm sure large companies have custom build tools.

We really do need something like github for electronics with features like
- wiki, issue tracker, maybe revision control (basically what github provides)
- Visual diffs for PCBs
- Smart inventory manager that can price check across distributors
- eCommerce integration for people that sell kits (what spanner888 was talking about)
- Social features like user project portfolios or public part lists [ my dev board is better than yours :) ]

Funny, this looks like what Upverter is trying to achieve except from a different angle.

http://demo.aligni.com (http://demo.aligni.com)

http://demo.aligni.com

Interesting. They import your parts into their online database which tracks projects through the different stages of its lifetime. You can keep track of your distributors and keep records of quotes for different parts. They store digital assets like datasheets in your Amazon S3 account if you have one. They also seem to do purchasing, but I'm not sure if they have relationships with the manufactures or if that is up to the user to deal with. From their doc:

Email a private ticket to view the RFQ to each contact.
Receive completed quotes from vendors and discuss them within Aligni.
Convert quotes from vendors directly to quotes within your database.

Sounds like they act as the middle man.

Overall, looks like a very solid inventory system. A bit confusing for me cause I'm probably not their typical user, but still impressive. Do you have experience with this BrentBXR?

Overall, looks like a very solid inventory system. A bit confusing for me cause I'm probably not their typical user, but still impressive. Do you have experience with this BrentBXR?

no I ahve seen it used and was very impressed by all its features. for my personal use i simply use:


I use a simple program called zparts. Google it; its on source forge. its no longer activly developed but it works.

It just simply allows you to make catagorys/sub catagorys and then differnt parameters for each sub catagory. So for example I have:

Microcontrollers:
 - > ATMEL
 - > Microchip
 - > TI

And for each part you can include all your own custom parameters (like flash size, pin count, quantity, and so on..) but it also allows you to link a picture and datasheet.

Looks like the Use cases discussed are:
   1. inventory of parts on hand
   2. parts required for project BOM (complete, or minus parts on hand) including cost of parts and shipping
   3. shopping cart(s) of project BOM optimised for cost, delivery speed, completeness (eg all/most/critical/some parts available)

Ian said:

One issue is its hard to get fuzzy descriptions to give good part suggestions. We tend to write descriptive BOMs because parts are interchangeable: "1uF 16v+ 0805", not manufacturer part numbers like "MDF-517-63409-1016A". Complete part numbers on parts like LCRs, in most cases, increases the price of our stuff because Seeed presumes they have to import that specific part.

This highlights the big challenge in sourcing parts - getting from a specification to a part you can order at the "right price".

The "right price" is the purchase$ and also the cost of ensuring availability, supplier/batch quality, correct footprint, total number of orders for a project is not excessive, as having to order a project BOM from several suppliers gets expensive as the shipping fees accumulate etc.

This is hard, it requires manual effort for every order, often a lot of effort because items are often out of stock or price / price break change, or your purchase goals are different to last time, or  your goals different to other people (eg shipping costs to your location often dictates the supplier, or quantity one off purchase/bulk, or fast delivery/cost no key factor....).

But once a BOM based on specs is published, the community can spread the effort and pool knowledge to source parts and keep the BOM + specs + supplier info up to date. Tools like those discussed here can really help that find part/substitute at the right $ sourcing process and the publishing of that info - Ian  said:

really like to see an open source, online inventory and project BOM manager

.

It seems highly unlikely that the commercial services are available to help the community at community pricing, so I would love to contribute to something like this. Are other people interested?

[quote author="nabilt"]I would like to know how small to medium size business deal with this.[/quote]

The size of the company does not matter too much, it's most the "amount of different stuff they build".

Bigger the company, more then tend to make "one thing" or they tend to don't build anything - they just send the schematic (often even without pcb layout) to be fabbed, they test and debug the prototype and then in few iterations get it done, without purchasing parts at all ...

I seen 4 different small companies that build a lot of small stuff (they make custom electronics for clients). And all four companies use exactly the same principle.

1. have a single cad app that everyone has to use (irrelevant what app they use, but 2 used altium, one used eagle and one used orcad) - noone is allowed to draw anything in any other cad.

2. they have a library of parts allowed to be used for design. Same library with sch and footprint details, datasheet etc etc on all instances of the cad tool they use (iirc altium can use shared libraries). ONLY parts in the library are allowed for projects. Parts are also labeled with priority "so you use  0805 if you can, then if you can't you can put 0603 and if that don't work you are allowed to use 1206 resistor... for example". If you need a part for your project that's not in the library it has to go trough "approval process". Person who's job is (among other things) that, check with all suppliers what "type" of that part they have so they check the availability options (how many is minimum order, how many usually available, on stock or not, how fast can it be delivered if off stock, money issues...) then they check EOL (how long device exist, is there a new generation available, what manufacturers are making it ...) and finally when they have all the data they add the part into library so from that moment anyone can use it.

I would like to know how small to medium size business deal with this.

Our ERP/management system is based on the slides SparkFun showed of their custom system called Sparkle at the sketching conference.

Also, we use an inventory management idea that Adafruit posted on their blog, a simple script addition to Zencart that adds all sales for each week for 90 days and recommends when to restock.

spanner888:
It seems highly unlikely that the commercial services are available to help the community at community pricing, so I would love to contribute to something like this.

I guess it depends on the business model. For example, github has done a great job of hosting open source projects for free and releasing parts of their backend like the wiki http://https://github.com/github. They convert a small percentage of their users into paying customers and we get better tools and hosting for git repos. Not a bad deal if you ask me.

spanner888:
Are other people interested?

I am. Like I said I'm planing to release the components that would be useful like inventory management, price checking etc, but I do plan on providing a hosted solution similar to github. So if someone else wants to put the components together to provide a similar hosted solution they would have to roll their own user authentication system and other stuff related to users. That said people might want to go a different way than I am planning. I'll help out either way.

Some background info. The app is currently built using the Pyramid framework AKA Pylons, MongoDB for storage, and dojo for the front end. It is fairly basic in terms of features at this point. You can make a project with parts and find the cheapest distributor. Right now I'm adding a search feature so the user can change the part if the system finds the wrong one.

arhi:
I seen 4 different small companies that build a lot of small stuff (they make custom electronics for clients). And all four companies use exactly the same principle.

Fascinating. It seems like it would be a pain to work with all of these restrictions, but that's life in the corporate world I guess. This also reinforces the need for a better online collaboration tools for hobbyists and companies like DP since it would be difficult to implement these policies on the internet.

ian:
Our ERP/management system is based on the slides SparkFun showed of their custom system called Sparkle at the sketching conference.

Cool. I didn't know Sparkfun open sourced this kind of stuff. Adafruit also does a great job in this area like you've mentioned. Cool to see you guys are all working together.

Sparkfun dos not open source theirs. That is why we started ours, to be the OS version, but also we have too much internal stuff hard-coded to make a public release (yet).

[quote author="nabilt"]
Fascinating. It seems like it would be a pain to work with all of these restrictions, but that's life in the corporate world I guess. This also reinforces the need for a better online collaboration tools for hobbyists and companies like DP since it would be difficult to implement these policies on the internet.
[/quote]

actually no. It makes the life for everyone way more simplified. You have r&d budget that you can use to source "any part" from "any source" to do testing, but when you start designing a project it can't happen that "this part is no longer available" or that you select a part similar to what company already have on stock but different enough so that you have to stock the new one too... so just like Ian uses "descriptive" BOM values (sot23-5 3V3 LDO for e.g.) the company actually finds the "best" 3v3 regulator and all projects use that one - they know where to order from, they have a stock as they use a bunch of them, they know it will exist in future, and they know all quirks of the part as they always order from same source .. it does require a person who spend some time on searching and organizing, but even in a 5 person organisation it's a no biggy to dedicate one person to spend time on this when need arises

This even makes sense in one person "organization". Really, how many different op-amps, regulators or processors you want to store, know how to use, have tools etc? If a part is just as good as another part, it is good sense to use the one you have and know.

Like I said I'm planing to release the components that would be useful like inventory management, price checking etc..... I'll help out either way.

Quote

This is a great offer, so when you are ready, just let us know what tasks would you like help with and we will chip in.

OK.

[quote author="nabilt"]Wow! That's amazing brian! Any reason you didn't use a part search engine like Octopart or Findchips? Also, how difficult has it been parsing the distributors websites? I find my self doing this for different projects and I'm always worried their website will change.[/quote]

Honestly I find that Findchips often fails to find chips that I know they have. Octopart seems better but still not always right.

Your concern about them changing their website is very valid. If they change the format of the invoice or the website my program needs to be updated. However given what I mostly care about is optimization of repeat orders I think it is a better way to make sure I get exactly the same part every time. Also it makes the project independent of anyone's API like Octopart. I think it would take a few hours at most to fix the parsers, since I will open source it hopefully people who are better programmers can keep the functionality but abstract the method a bit more?

As far as how difficult the websites are to parse: Not very. Everything is just generated by a script on some database so you just have to do the reverse of that script to get back to the data.

I am basically working on this program (called teho Logistics), MSPism and openHiFi software wise ATM. Hardware wise I have 4 designs simmering and I have a lot of real work also, thus progress is slow. teho Logistics gets time when I have large blocks of dead time in the clean room as I can code while I wait on my laptop. Hardware stuff only gets done at home.

Not everyone will like how I do things, it is very unix, IE files and not GUI. Because everything is a CSV you can just use a spreadsheet if you need to edit the data enmass.

I found a command line project from Techo Labs in the early stages with some interesting features.

http://teholabs.com/2012/01/tehologistics/ (http://teholabs.com/2012/01/tehologistics/)
Source: https://github.com/teholabs/tehoLogistics (https://github.com/teholabs/tehoLogistics)

From the readme

Features:
Imports invoices Mouser PDF, Digikey PDF, Arrow HTML (email)
Keeps current inventory and average unit cost of application identical parts in CSV file
Removes inventory via "building" lists (AKA making PCBs), lists defined externally via CSV file
Look up and modifiy inventory data amount of single parts
Check if parts are on hand for production of set of designs and create shopping list if needed
Create buy lists for lowest cost source importable at Arrow, Mouser, Digikey parts needed a production run

@nabilt

That is the project I mentioned in this thread already ;-)

Also I made a forum post here about it: viewtopic.php?f=2&t=3385 (http://dangerousprototypes.com/forum/viewtopic.php?f=2&t=3385)

as I don't run a forum at teho Labs at this time.

Ohhh. Silly me. I didn't put the two together and it was late...

No worries, good to know someone reads the google+ posts

interesting thread :-)

lol as a noob, i use a big cardboard box for my parts and few paper wishlists  :oP
lol

[quote author="nabilt"]We really do need something like github for electronics with features like
- wiki, issue tracker, maybe revision control (basically what github provides)
- Visual diffs for PCBs
- Smart inventory manager that can price check across distributors
- eCommerce integration for people that sell kits (what spanner888 was talking about)
- Social features like user project portfolios or public part lists [ my dev board is better than yours :) ]

Funny, this looks like what Upverter is trying to achieve except from a different angle.[/quote]

Hm, I really want to build something like this or at least be part of creating this :)

P.S. About price check across distributors - I believe that it is not possible to reliably automate this check, because some distributors may have multiple entries of similar items (as digikey), or may have mistakes in descriptions (as mouser), or too general classification and not complete on-line description (as jameco). I think it is important to have actual human experience in ordering something from some partictular distributor. For example jameco has listed one 20v8 device, but actually they have GAL and PALCE devices under the same id (and they are incompatible from programmer point of view - I must specify which one I want in moment of ordering). Also jameco has 2 entries for 0.1uF tantalum capacitor and only paper catalog says that one has 0.1' distance between contacts and another has 0.2' etc. So ideal solution for this problem is kind of social network of hobbyists who shares their own experience about some particular items that they bought from one or more distributors. Also it is really interesting to know packaging habits of distributors as tendency to put a lot of papers in the box and/or actual shipping pricing for different states etc.

About price check across distributors

Octopart has an API that can help access some of this info. The fuzzy match thing is still an issue though.

Shaos:
Also jameco has 2 entries for 0.1uF tantalum capacitor and only paper catalog says that one has 0.1' distance between contacts and another has 0.2' etc. So ideal solution for this problem is kind of social network of hobbyists who shares their own experience about some particular items that they bought from one or more distributors.

I really like the idea of a social network helping you choose parts. One of my ideas was to have a public parts list so we can share knowledge about our favorite parts. For example, maybe you want to find a pressure sensor that is reliable and has an easy interface. The user could do a search and find the parts list for an open source project like the arducopter, which uses the MEAS MS5011. That might be a good place to start researching since they have presumably tested a few.

One of my ideas was to have a public parts list so we can share knowledge about our favorite parts.

You're more than welcome to put your favorite stuff on our partlist, or expand it with a new section:
http://dangerousprototypes.com/docs/Partlist (http://dangerousprototypes.com/docs/Partlist)

I was wondering, if that part search could be one plugin per distributor, so it would be easy to add local shops to the program without annoying people around the world. Also each plugin could use their own thread and get the search done faster? Just my cents.

If I understand correctly you want a way to add distributors that are not indexed by Octopart like the local electronic shop. This is a good idea that I could see adding in the future. People could write a plugin to search their particular distributor, but you will have to enter this information manually for now.

Octopart just tweeted a free tool for BOM management. You can import a CSV file into a project and auto fill details using Octopart.
http://http://www.arenasolutions.com/partslist/?utm_source=Twitter&utm_medium=tweet&utm_term=arenapartslist&utm_content=ECAD%2BBOM&utm_campaign=plpromo

Their main tool is a more comprehensive product with collaboration features and stuff. They charge $79 per user per month, which includes training, tech support, upgrades, etc.
http://www.arenasolutions.com/ (http://www.arenasolutions.com/)

Engineer from octopart here, hope that doesn't step on any toes...

If we can all come up with a good general purpose structured data format, it could be used for interchange between EDA tools, inventory management software, lookup and order placement APIs, etc. One common anti-pattern we see is designers/purchasers blurring the line between part descriptions ("1k 0805 2% resistor"), manufacturer part numbers, and distributor SKUs. Certainly each has their place, and any of these individually or in combination is a valid line item that can be searched for, but not in the same data field please!

We are designing our next iteration API over the next few months, any feedback on the existing one would be great. We have existing customers using the v2 API, so we will continue to support that interface as well for the foreseeable future. We have a number of rate-limiting measures in place to prevent what we see as abuse (scrapping our entire database of millions of parts), but certainly want to support use cases like checking for price updates on dozens or hundreds of parts in a reasonable time frame.

Anybody have a favorite existing interchange format to replace .csv? Something flexible enough to be used across octopart, upverter, solderpad, and potentially get sent off to contract manufacturers?

Welcome to the forum!!

/me votes for CSV !!

dBase perhaps? Its nice because its not an excel or office format but excel/access does support it. There are a couple different dBase versions but they are basically spreadsheets with headers.

A more user/software friendly choice would be XML, as all major and even minor software and web languages usually have a built-in support library for XML parsing... reading and writing.

You can import CSV into ANYTHING (directly into MySQL/PGSQL/EXCELL/OpenOffice/LibreOffice/Access...) and you can parse it in any language blindfolded. dBase to parse you need to use 3rd party libraries that are last used 15+ years ago. XML adds huge overhead and provides less to none benefit (yes you can have format-less data - do you really want that?!), JSON is very popular these days but what would be the point of sending JSON around as again it's only going to create bigger and harder to parse file.

I think it's really a no-brainer :D

XML is ubiquitous, JSON is nicer to read but, really, CSV makes the most sense.  It's more efficient than XML or JSON, still human-readable, and a lot of people are currently using it in their parts workflows.  Seems a no-brainer.  (edit: nod to arhi for saying it first)

I dislike CSV's awful standards as much as anybody but, really, you gotta use the best tool for the job.

I guess I read the OP request a little differently.  Not so much what the file format would be but what the data model for a parts object would be.  That is, what are the fields and what do they mean so that the different web sites and programs can try to exchange information effectively.  I don't really have an idea about this to contribute unfortunately, but I can certainly see it is a problem whenever I contemplate an electronic inventory of my parts collections.  Just trying to collect together the information from the various suppliers is not easy at this point.

Oh, and one thing about CSV format.  Unless there is something about the CSV standard that I don't know (certainly possible) it seems to me that the biggest downside of CSV, compared to XML or JSON, is that CSV is basically a model of a 2D table, without any clean way to represent hierarchical data.  If your data fits nicely into flat rows, then great.  Otherwise, you need a more structured format, it seems to me.

You can fit a multi dimensional array in csv but the question really is if you need one. If you do then another format is better. Thing is thinking hard I can't see a reason for multi dimensional / hierarchical data for the parts.

The "library in your EDA" need hierarchy. The simplest example is "resistor" can be 100W, 10W, 2W, 1W, 1/4W, 1206, 0805.. This is also not stored in some muti dimensional array but in simple referenced tables... now if you want to share it and want to send the data in single file then CSV would not nearly be the best option - but this is not the data you want to send. As I understand the project we are talking about "physical part" that you will source from somewhere. So It can't be 100W or 0805, you want exactly "this" resistor and that resistor don't have any "multi values" (except order source and price but that you do want to have in reference table and not in the same table as that resistor)

@arhi, exactly, there multiple contexts in which parts lists/BOMs are passed around, and thus multiple expectations for what a "line item" consists of. That's fine, but it would really help if these files were annotated better to reduce confusion.

I think the most common two contexts are "order-able BOMs" which are pretty much what get split up and submitted to distributors, and a design component lists, which have a description for each circuit element ("U2", "R12", etc) in a design. The former is more likely to have specific manufacturer parts specified (though maybe not for passives) with summarized quantities, while the later is more likely to have general descriptions (ambiguous what exact part would be used) and duplicate line items.

Could one file format handle both cases? Uncertain, if multiple different MPNs are acceptable for a single circuit element (eg, 555 timers from different manufacturers), then there is a many-to-many relationship between order-able parts and circuit elements. Do we need two file formats, a relational file format with guarantees (eg sqlilte, not hand editable) or should we nest one "line item" type within another (aka, list of order-able parts each with a list of circuit element identifiers, or a list of circuit elements with a list of acceptable order-able parts)?

At a minimum I think it's important to stop blurring the line between manufacturer MPNs and distributor SKUs. Both can be unique identifiers for an order-able part, which is great (and I don't want to tell anybody they shouldn't use SKUs because a lot of MPNs are bad or ambiguous), but when they are used interchangeably without annotation it makes it very difficult to do efficient lookups, which makes it hard to build for anybody to build awesome tools to help share awesome designs and make the boring parts of electronics design and production go away. The adafruit and dangerousprototypes part lists are great because they distinguish between SKUs and MPNs.

I think blog/forum posts on this topic in the Python PEP / internet RFC style would probably be the best way to put forward proposals.

PS, thanks for the blog post!

"CSV" doesn't specify anything at all. Spreadsheets can handle them flexibly, but there is no standardization: comma separated with single quotes? tab delimited with no quotes? ASCII with escapes? UTF-8? Chinese characters? International currency symbols? Ohm symbol? What kind of new lines? Can you write a program to determine if an arbitrary input fits the specification? Or even reliably detect the character formatting in a cross-platform way? Line-item CSV import/export tools are very important for any service to allow editing by spreadsheet, and maybe people will keep maintaining their BOMs that way forever, but CSV really isn't acceptable for communications between tools and services.

JSON is slightly better, is human grokable and editable, and handles nested hierarchies. We've used Thrift internally and like it, protocol buffers are also nice. XML has a lot going for it. Personally I think a JSON micro-format is the best choice, but XML is more polished as an interchange format, and a formal document spec might save headaches down the line.

Here are some example JSON chunks; I recommend using a browser extension like JSONview to inspect these:

Example solderpad.com BOM ('designator', 'description', 'value'):
http://solderpad.com/solderpad/arduino- ... ty_print=1 (http://solderpad.com/solderpad/arduino-ethernet/bom.json?pretty_print=1)

Example octopart.com part (complicated):
http://octopart.com/api/v2/parts/get?ui ... ttyprint=1 (http://octopart.com/api/v2/parts/get?uid=39619421&prettyprint=1)

Upverter has "instances" ('instance_id', 'library_id' pointer, attributes) and "components" ('library_id', 'name'):
http://upverter.com/resources/open-json-format/ (http://upverter.com/resources/open-json-format/)

Can anybody post an Eagle XML or rEDA example? KiCad looks messy.

I think any 'database' or export of this nature its quite important to be able to develop a multi-level file; not just a simple spreadsheet. at least in imho it offers the most ability for upgrades in the future as well. CVS is old school and doesnt seem to be flexible. Using an XML file you could easily develop a website for example and only read certain parts of the XML file you are seeking for example; with CVS you basicly have to load everything into memory; ditch what you dont want then show what you want... Im sure theres ways not to ect ect... ect.... do dont bother arguing that point :) im just saying its easyer with XML you really dont have to do much code at all to get it working.

I dont know; imo i think xls/json is the way to go.

It is called XML and is an industry standard for data interchange.
Most other systems are not hierarchal, with XML you define the 'outer case', then inside that you nest the specifications of the parts, in a tree like fashion, then it does not matter how you define the part , or even that you need a standard 'field' for part number, you let the XML self define the part.

wrt csv, the format will be "whatever you decide to make standard", it's not hard to define it ... most tools will import any but some classic is
 - utf8 (with or without bom code)
 - tab or comma separator
 - decimal point for floats
 - normal double quotes  (") for string delimiters and standard escape (") to escape them
 - n for line end

that works for any system (easy to import/export)

now, wrt relations, just like as you can store data in a relational database you can export each of those tables as a simple csv file that one can then use. This way for e.g. you have your "manufacturer" table (motorola, nxp, microchip, atmel, TI...) that one do not need to download from you "every time", but sync with you only when table get new rows (you should never delete a row from a table!!), a "reseller/source" table (farnell, digikey ...) again fairly static and slow changing but big that again one fetch new version from time to time .. and then you have tables describing parts (classes, categories etc etc) where some tables are again fairly static while some will be dynamic and finally a "part" table that's dynamic table holding "each physical part you can identify" that one should never "fetch whole" but "query only" trough some interface.

That would be the simples / fastest way as the data transfer between systems is minimal, and load on query servers is reduced as much as possible (you search trough fairly small table as it consist only number references and few number values so it's indexable, cacheable ...) but requires more work on the client side.

You can of course go to "simple client" philosophy where you have dummy clients that hold no data whatsoever locally and require a full result set to be generated by you. In that case the info you return has to be multidimensional and descriptive hence you can't use simple overhead less format like csv and you have to go to higher level response type. XML was a great idea but the implementation is where it got completely screwed. The overhead is huge, parsers are written mostly by idiots and the limitations are incredibly high, especially for us where in electronics you really need to use "special characters" hence all data would have to be inside CDATA blocks hence none of the "readability" we are talking about would remain. On the other hand, JSON parsers are weirdly written pretty good (mostly by same guys who wrote xml parser ?!) they work pretty nice and the protocol don't add too much overhead while keeping the required features (multidimensional and descriptive format).

It's really up to you how you want to proceed. In order to generate reply to a question "gimme who has part with model 7 make 12 and option 22" with 7,10,22,22,12,7.3n7,13,12,22,12,7.5 .. or you want to "find me all *555* devices" and you need to create a json structure with all the strings naming the manufacturer, model.... it will take 100 times more resources on your side, but will be waaaaaay easier for the client

[quote author="hardcore"]It is called XML and is an industry standard for data interchange.[/quote]

by what industry? I work for company that invented XML and we don't use it except in few "still unchanged" systems and those will be changed too in near future. XML is removed from almost any modern system. On the WEB, and I assume this is what the collaborative parts management will be using as a base/technology, XML is completely replaced by JSON, and to be honest, XML never really fully lived on the web. Not that Sun and W3C didn't do their best to promote XML, XSL/XSLT, more layers, more complexity to achieve simplicity etc etc, they invested huge amount of money and sweat and blood (ours) to get XML to be "the industry standard" but actually - it never became so...

On the other hand, if you are using web app, there's a 99% chance it's sending JSON packets back and forth...

I don't really prefer any format over the other, json has better parsers and less overhead, xml looks nicer, I personally dislike both and I personally use both every day, and when someone say XML is "human readable" it makes me cry..

I vote for XML/JSON with a slight edge for JSON cause it would make life on the web easier. I like the flexibility and self documenting structure these formats provide. However, arhi's point about overhead is a valid one. Parsing JSON that Octopart returns has become the bottleneck for my web app. I still have to experiment with other libraries and possibly multithreading to see if I can speed things up.

Other formats that might be worth researching:

Product data eXchange
Arena uses this, but I've never heard of it - http://www.arenasolutions.com/pdxviewer/ (http://www.arenasolutions.com/pdxviewer/)

Since Eagle moved to XML it would be interesting to see what schema they use. Do they add the data they find from Design Link into the BOM? http://www.adafruit.com/blog/2011/11/04 ... reenshots/ (http://www.adafruit.com/blog/2011/11/04/eagle-v6-beta-xml-export-some-example-files-and-screenshots/)

Altium also has their Live Links to supplier data. I'll investigate to see what format they use as well.
http://altiumwiki.onconfluence.com/disp ... plier+Data (http://altiumwiki.onconfluence.com/display/ADOH/Live+Links+to+Supplier+Data)

Shouldn't we sort out exactly what data is required in the BOM before worrying about the data format?

I think a huge shortcoming of most current BOMs is they lack all or proper footprint information. It is very difficult to use most current BOMs to purchase parts and have confidence that ALL the parts supplied by different suppliers will fit that original PCB design!

Another important step is to see what others are doing in the BOM and related space.

I got carried away with some justifcation/background of this below, so feel free to skip my ramble and look at the example links at the end.

A simplistic process is :-

   Design - source - build

This is not very realistic, as "real" projects have many approaches and include through various prototype, test, release, maintainance, forking, ... stages so versioning is also important. Plus sharing of data at every stage is important.

I chose these to give a focus for the discussion, because of the different information about parts that is required during each of these stages.

Very simplistically, the focus at each of these three steps is:

   Design - specifications, footprints, pricing
   Source - all above + quantity, preferred suppliers, .....
   Build  - labelling info (so you know which thing is what), footprints (so you know how to put the thing in the PCB)

Reality is that these are not seperate, independant steps, but interwoven, as design includes PCB layout - so you need the part footprints, and design also includes cost, so you need sourcing information then as well. Build includes many production runs, so that requires sourcing many times (which means changes to suppliers, parts, footprints, and design revisions...). All of this still applies to the hobbyist, but the terminology or process may be different. Even things like environmental information (ROHS...), material handling info is important to the hobbyist to keep themselves and the parts safe!

The design and sourcing steps are particularly challenging as there is ongoing "conflict" with the parts required in the design, the pricing the source and the footprints.

The footprints are particularly important because they:
   - impact the design - PCB layout
   - sourcing - who can supply the part with THAT footprint
   - build - does the part ACTUALLY fit the PCB
   - any change to a footprint impacts ALL steps!!!
   - it takes a LOT of effort to maintain the footprint information in your parts library (Eagle, Kicad, ...) and for your (haha) pick and place machine.
   
Footprints are important at EVERY stage, yet they typically only get mentioned in a BOM as TO22, or SOIC or DIP, yet that is incomplete as it does not give pinouts for that device, nor does it cater for the myriad of switches and connectors and ......

It also does not include your own custom keepout/spacing that might be required for that project, and a myriad of other specifics of a part from EACH manufacturer of the "same" part!

Because of the issues that continually occur with footprints, nearly "everyone" keeps there own trusted and tested part library (ie with footprints), or uses a trusted library from someone.

So they are really one critical part of the design information and should be included in BOTH the part library AND the BOM of the project
as well as the gerbers(?) for pick and place. The footprints need to be specified in detail and to enable others to source and build a project must be included in the projects PCB information and the sourcing (BOM) information.

Then there are so many current design tool data formats, and the vast amount of manufacturer part data, which is "resupplied" via the suppliers.


There are a lot of people working around these issues, for example:

http://pcdandf.com/cms/designnews/8308- ... cb-library (http://pcdandf.com/cms/designnews/8308-a-one-world-standard-pcb-library)

http://www.pcbstandards.com/forums/show ... php?t=1677 (http://www.pcbstandards.com/forums/showthread.php?t=1677)

Unified CAD-PLM Architecture ..... http://nesl.ee.ucla.edu/fw/documents/co ... c-apex.pdf (http://nesl.ee.ucla.edu/fw/documents/conference/2011/friedman-friedparts-ipc-apex.pdf)

http://folknologylabs.wordpress.com/ (http://folknologylabs.wordpress.com/)
   their idea = bom + search across vendors to give MULTIPLE shopping baskets for EASY PURCHASE
   started playing https://github.com/folknology/iBom (https://github.com/folknology/iBom) - just a sample BOM format

http://usabledevices.com/category/parts/ (http://usabledevices.com/category/parts/) - a related and of course highly relevant idea (mine!)

The reply above in this thread by bnewbold from Octopart provides a different perspective on the scale of this challenge.

Hi,

[quote author="bnewbold"]Engineer from octopart here, hope that doesn't step on any toes...

If we can all come up with a good general purpose structured data format, it could be used for interchange between EDA tools, inventory management software, lookup and order placement APIs, etc. One common anti-pattern we see is designers/purchasers blurring the line between part descriptions ("1k 0805 2% resistor"), manufacturer part numbers, and distributor SKUs. Certainly each has their place, and any of these individually or in combination is a valid line item that can be searched for, but not in the same data field please!

[/quote]

As most part definitions have multiple distributors with different naming schemes we are using internally for our projects a type based naming of parts.

The general rule is the following:

[tt:]<type>-<package>[-<pincount>]-<value>[-<additional specs>][/tt:]

Here are some examples:

[tt:]res-0402-1k5
res-0603-44-0.25w
cap-cer-0402-1u
ic-so8-can_transceiver
mcu-tqfp-32-stm32f105
vreg-sot23-5-3v3[/tt:]

We use those ID's to link our PCB designs with part manufacturer selection, project BOM price calculation and final purchasing and assembly, using bom-x in Eagle and a damn big and complicated google docs spreadsheet. :)

It would be nice if we could just drop an ID like [tt:]res-0402-10k[/tt:] into Octopart and get a full list of 0402 10k resistors and find the cheepest source for it. Important to note is that in such a search we would not care about tolerance and wattage.

Things become more complicated with for example capacitors where you may care about which dielectrics you _don't_ want, for example those that start with X and Y. I am not sure how that should be handeled.

I still would love some consolidated standard way of naming electronic parts, so that it is easier to find the right manufacturer and supplier.

Cheers Esden

[quote author="esden"]
It would be nice if we could just drop an ID like [tt:]res-0402-10k[/tt:] into Octopart and get a full list of 0402 10k resistors and find the cheepest source for it.[/quote]
That is what I am trying to do with my web app. It tries to extract useful data from the package, value and description and finds the cheapest part. I had the same problem with existing tools because they expect your BOM to have exact part numbers before they do a search. Eagle's Design Link tries to solve this problem, but it doesn't do a very good job.

[quote author="arhi"][quote author="hardcore"]It is called XML and is an industry standard for data interchange.[/quote]

by what industry? I work for company that invented XML [/quote]

So you worked for  the W3C or Sun microsystems back in 1996, and that makes you an expert on the subject how?

'96 has nothing to do with it, TODAY the XML is being removed from industry, and TODAY even the company that invented it don't use it any more .. the whole web community basically ditched the XML for JSON. I personally don't find any of them "better", I was just stating the facts.... just like 1wire in electronics, XML is available only in legacy apps and is being pushed out very fast..

As for being expert, that's completely irrelevant, take this as my personal opinion only..

@esden, there is a similar project (from 1994!) at http://partnumber.com/; (http://partnumber.com/;) it creates a terse "generic" part number given a category, value, and variance. I find your examples much more readable and flexible, but i'm skeptical any formal scheme like this will gain widespread use. Much more likely is that human-readable conventions around shorthand will develop and be combined with "smart" full text search engines. For example, here are some full text searches with some of the examples you gave, slightly paraphrased:

http://octopart.com/partsearch#search/r ... 02+1.5kohm (http://octopart.com/partsearch#search/requestData&q=resistor+0402+1.5kohm) (30 hits)
http://octopart.com/partsearch#search/r ... 3+44+0.25w (http://octopart.com/partsearch#search/requestData&q=resistor+0603+44+0.25w) (no results)
http://www.mouser.com/Search/SearchResu ... 5w&FS=True (http://www.mouser.com/Search/SearchResults.aspx?Keyword=resistor+0603+44+0.25w&FS=True) (no results)
http://octopart.com/partsearch#search/r ... t23+5+3.3v (http://octopart.com/partsearch#search/requestData&q=voltage+regulator+sot23+5+3.3v) (312 hits)
http://www.mouser.com/Search/Refine.asp ... t23+5+3.3v (http://www.mouser.com/Search/Refine.aspx?Keyword=voltage+regulator+sot23+5+3.3v) (138 hits)
http://octopart.com/partsearch#search/r ... +stm32f105 (http://octopart.com/partsearch#search/requestData&q=mcu+lqfp+32+stm32f105) (6 hits; lqfp not tqfp!)
http://www.mouser.com/Search/Refine.asp ... +stm32f105 (http://www.mouser.com/Search/Refine.aspx?Keyword=mcu+lqfp+32+stm32f105) (8 hits; lqfp not tqfp!)

That being said, if you do have a formalism of your naming scheme (aka, definitions of all the various type abbreviations and values; "cap-cer" doesn't fit the scheme) i'd be happy to whip up a parametric search front-end for octopart.

Has anyone tried ecdb.net ?? looks good. :)

I did. Looks like he is off to a good start. Here is the hackaday post http://hackaday.com/2012/03/12/ecdb-off ... organized/ (http://hackaday.com/2012/03/12/ecdb-offers-a-simple-intuitive-way-to-keep-your-electronic-inventory-organized/)

I manage my parts by simply searching the site: Component Search, for the parts I currently need, with a small supply of those I use the most always kept in easy access.