Search Github projects by component, find design references

Posted on Thursday, November 12th, 2015 in components by DP



Search Github hardware by component to find examples and references for your next project. Big schematic and PCB previews mean quick and easy browsing without opening a CAD program.

TomKeddie first mentioned this idea at Hacker Camp Shenzhen, and later in the forum and on WeChat. Tom generously shared his scraping/search method. Eagle 6+ files are XML, so we can find them on Github by searching for the “eagle SYSTEM” tag in files with “extension:sch”. That gives more than the maximum 100 pages of results, so we filter by file size and increment size 500 bytes at a time “size:1001…1500”. We use the normal user search interface, parse the HTML results, and grab all urls ending with .sch. While Github has an API, that API doesn’t give access to search code search without specifying a repository by name (probably so people don’t do what we did…).

There are a number of limitations. Only Eagle projects are indexed, a search for KiCad files inspired little interest in expanding. Before Eagle 6.0 files were not XML and will not be included. Github search only indexes code files up to ~390K, larger files will not be included. If Github improves access via the API we will expand the index.

Scraping all of the files over terrible Chinese internet took about a week. Searches were performed at minimum 32 seconds apart. Some files were incomplete (lacked closing tag). Some files have extra ‘>>’, usually a merge error tag. Some files were just corrupt. These all caused Eagle to hang until the window was manually closed. Working with Eagle from the command line has been most unpleasant.



Via the forum.



This entry was posted on Thursday, November 12th, 2015 at 11:01 am and is filed under components. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

One Response to “Search Github projects by component, find design references”

  1. J. Peterson says:

    This is very cool. It’s probably worth noting that Github being what it is, you may be looking at somebody’s half-baked work in progress. But even that can still beat starting from scratch or answer questions the spec sheets don’t.

Leave a Reply

Notify me of followup comments via e-mail. You can also subscribe without commenting.

Recent Comments

  • KH: And that looks really expensive... Only browsed the vid though, I'm an dinosaur so I had the sound off too. Nice of him to open-source...
  • Anton: I really need bus pirate!! Thanks
  • Crawford: Happy Father’s Day!
  • Fred Fish: me
  • Keith: I'd love one.