Module 1: Reproducible research toolkit introduction

Science is facing a crisis where error, omission, and fraud are threatening the quality of evidence we rely on to make decisions like which medical treatments to use and which social programs to fund.1–3 One evidence-based strategy to improve the quality of scientific research is the adoption of reproducible research practices.

Using reproducible research practices speeds up scientific discovery, fosters greater exchange of ideas among scientists, and reduces research waste.4,5 Research papers with shared data have fewer errors, those with shared data or shared code are cited more, and shared data are used in more publications.5,6

The modules in this toolkit will guide you through building a reproducible research project. Pastry chefs know that to re-create that most excellent souffle, they must have the same ingredients and a clear and complete recipe to follow. Likewise, to reproduce scientific research, scientists need access to data and a clear and complete recipe for data management and analysis.4,7,8 After this brief introductory module, the other four modules in our toolkit will help you learn:

  • Module 2: Strategies to make private data shareable
  • Module 3: How to prepare statistical code to share
  • Module 4: How to document data and code for sharing
  • Module 5: How to share data and code on GitHub

Examples of toolkit use

We invited several researchers to help test and improve the toolkit. The researchers used the toolkit to format, organize, and post materials from a recent publication. The GitHub repositories for these projects are available as examples:

For best results, we suggest reviewing each module in its entirety before implementing the changes. Use our reproducibility checklist to keep track of your progress as you go through the modules.

If you want to know more about reproducible research, there are many other publications9–19 and online resources available.

Software and data file formats

We know that everyone has a favorite, go-to statistical software platform. Throughout the modules, we provide reproducible examples using four platforms popular in public health: R, SAS, SPSS, and Stata. We hope that this will make our modules useful and accessible for everyone.

One overarching recommendation we make is the use of non-proprietary file formats when making data and code public. In all or most platforms, data can be saved or exported as a comma-separated values (CSV) format, and in many, code or codebooks can be saved as a PDF or native format. If you use a platform where codebooks cannot be saved out in a non-proprietary format, see the examples in Module 4: How to document data and code for sharing. These types of files (e.g., CSVs, PDFs, TXTs) and their kin can then be imported back into most other platforms so that end users can reproduce work in their preferred software. We have two main motivations for promoting the use of non-proprietary formats for reproducibility: 1) Using non-proprietary file formats increases accessibility to code and data and widens the audience of potential reproducers of your awesome work; and 2) Money. If you are no longer a student, think back to when you found a perfect dataset, but did not have access to Stata or SAS, and couldn’t even think of buying it.

The coding2share project

The coding2share project was funded by the Robert Wood Johnson Foundation’s “Open Access” strategy. In the first part of the project we surveyed 200 public health scientists about their current reproducible research practices. The goal of the survey was to gain an understanding of current practices and desired training for reproducible research, and to use what we learned to design these modules.

One thing we found was that dissemination of data and code was rare. The most common thing researchers reported sharing was clean data, and even that was only shared by 12.4% of participants.

About 25% of participants reported not having a codebook for a recent data project and the use of recommended coding practices was not even close to universal:

Survey participants reported that data privacy was the most common barrier to making data or statistical code publicly available.

Survey participants also thought that training, requirements, and incentives would facilitate use of more reproducible research practices:

More detailed results, data, and statistical code will be made available in our GitHub repository shortly.

Resources

References

1. Fang FC, Steen RG, Casadevall A. Misconduct accounts for the majority of retracted scientific publications. Proc Natl Acad Sci USA. 2012;109(42):17028-17033.

2. Steen RG. Retractions in the scientific literature: Is the incidence of research fraud increasing? J Med Ethics. 2011;37(4):249-253.

3. Prinz F, Schlange T, Asadullah K. Believe it or not: How much can we rely on published data on potential drug targets? Nat Rev Drug Discov. 2011;10(9):712-712.

4. Peng RD, Dominici F, Zeger SL. Reproducible epidemiologic research. Am J Epidemiol. 2006;163(9):783-789.

5. McKiernan EC, Bourne PE, Brown CT, et al. How open science helps researchers succeed. eLife. 2016;5:e16800.

6. Piwowar HA, Day RS, Fridsma DB. Sharing detailed research data is associated with increased citation rate. PloS One. 2007;2(3):e308.

7. Peng R. The reproducibility crisis in science: A statistical counterattack. Significance. 2015;12(3):30-32.

8. Towards transparency. Nat Geosci. 2014;7:777.

9. Ali-Khan SE, Harris LW, Gold ER. Motivating participation in open science by examining researcher incentives. eLife. 2017;6:10.7554/eLife.29319.

10. Anderson CJ, Bahník Š, Barnett-Cowan M, et al. Response to comment on Estimating the reproducibility of psychological science. Science. 2016;351(6277):1037-1037.

11. Begley CG, Ioannidis JP. Reproducibility in science: Improving the standard for basic and preclinical research. Circ Res. 2015;116(1):116-126.

12. Camerer CF, Dreber A, Forsell E, et al. Evaluating replicability of laboratory experiments in economics. Science. 2016;351(6280):1433-1436.

13. Goodman S, Greenland S. Assessing the unreliability of the medical literature: A response to “why most published research findings are false” 2007. Johns Hopkins University, Department of Biostatistics. Available at: http://www.bepress.com/jhubiostat/paper135. Accessed 1 December 2017.

14. Hirschhorn JN, Lohmueller K, Byrne E, Hirschhorn K. A comprehensive review of genetic association studies. Genet Med. 2002;4(2):45-61.

15. Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2(8):e124.

16. Ioannidis JP. How to make more published research true. PLoS Med. 2014;11(10):e1001747.

17. Laine C, Goodman SN, Griswold ME, Sox HC. Reproducible research: Moving toward research the public can really trust. Ann Intern Med. 2007;146(6):450-453.

18. Collaboration OS. Estimating the reproducibility of psychological science. Science. 2015;349(6251).

19. Peng RD. Reproducible research in computational science. Science. 2011;334(6060):1226-1227.

LS0tDQp0aXRsZTogIk1vZHVsZSAxIg0Kb3V0cHV0Og0KICAgDQogIGh0bWxfbm90ZWJvb2s6IA0KICAgIGNvZGVfZm9sZGluZzogbm9uZQ0KICAgIGZpZ19jYXB0aW9uOiB5ZXMNCiAgICBkZl9wcmludDogdGliYmxlDQogICAgdG9jOiB5ZXMNCiAgICB0b2NfZmxvYXQ6IA0KICAgICAgY29sbGFwc2VkOiB0cnVlDQogICAgICBzbW9vdGhfc2Nyb2xsOiB0cnVlDQogICAgaW5jbHVkZXM6DQogICAgICAgYWZ0ZXJfYm9keTogRzovQ1BIU1MvT3BlblNjaWVuY2UvTW9kdWxlcy9TdHlsZXNoZWV0cy9Gb290Lmh0bWwNCmJpYmxpb2dyYXBoeTogRzovQ1BIU1MvT3BlblNjaWVuY2UvV3JpdGVVcHMvYWltMVN1cnZleVBhcGVyL1JlcHJvU3VydmV5UGFwZXJfMDIxMjIwMTguYmlidGV4DQpjc2w6IEc6L0NQSFNTL09wZW5TY2llbmNlL1dyaXRlVXBzL2FpbTFTdXJ2ZXlQYXBlci9hbWVyaWNhbi1tZWRpY2FsLWFzc29jaWF0aW9uLW5vLXVybC5DU0wNCmVkaXRvcl9vcHRpb25zOiANCiAgY2h1bmtfb3V0cHV0X3R5cGU6IGNvbnNvbGUNCi0tLQ0KDQo8aGVhZCB7I3RvcH0gPiANCg0KPHNjcmlwdCBzcmMgPSAiRzovQ1BIU1MvT3BlblNjaWVuY2UvTW9kdWxlcy9TdHlsZXNoZWV0cy9NYWluLmpzIj48L3NjcmlwdD4NCjxsaW5rIHJlbD0ic3R5bGVzaGVldCIgdHlwZT0idGV4dC9jc3MiIGhyZWY9Ikc6L0NQSFNTL09wZW5TY2llbmNlL01vZHVsZXMvU3R5bGVzaGVldHMvTWFpblN0eWxlLmNzcyI+DQpgYGB7ciwgZWNobz1GQUxTRX0NCmh0bWx0b29sczo6aW5jbHVkZUhUTUwoIkc6L0NQSFNTL09wZW5TY2llbmNlL01vZHVsZXMvU3R5bGVzaGVldHMvTmF2YmFyLmh0bWwiKQ0KYGBgDQo8aW1nIGNsYXNzPSJsb2dvIiBzcmMgPSAiRzovQ1BIU1MvT3BlblNjaWVuY2UvTW9kdWxlcy9jMnMtbG9nby1mb3ItbW9kdWxlcy5qcGciPiANCiAgIA0KPC9oZWFkPg0KDQpgYGB7ciBzZXR1cE9wdHMsIGVjaG89RkFMU0V9DQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobz1GQUxTRSwgZmlnLndpZHRoPTEyLCBmaWcuaGVpZ2h0PTgsIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0UpDQoNCmBgYA0KIyBNb2R1bGUgMTogUmVwcm9kdWNpYmxlIHJlc2VhcmNoIHRvb2xraXQgaW50cm9kdWN0aW9uIA0KDQpTY2llbmNlIGlzIGZhY2luZyBhIGNyaXNpcyB3aGVyZSBlcnJvciwgb21pc3Npb24sIGFuZCBmcmF1ZCBhcmUgdGhyZWF0ZW5pbmcgdGhlIHF1YWxpdHkgb2YgZXZpZGVuY2Ugd2UgcmVseSBvbiB0byBtYWtlIGRlY2lzaW9ucyBsaWtlIHdoaWNoIG1lZGljYWwgdHJlYXRtZW50cyB0byB1c2UgYW5kIHdoaWNoIHNvY2lhbCBwcm9ncmFtcyB0byBmdW5kLltAUmVmV29ya3M6MjE4MjtAUmVmV29ya3M6MjE5MjtAUmVmV29ya3M6MjE4NF0gT25lICpldmlkZW5jZS1iYXNlZCBzdHJhdGVneSogdG8gaW1wcm92ZSB0aGUgcXVhbGl0eSBvZiBzY2llbnRpZmljIHJlc2VhcmNoIGlzIHRoZSBhZG9wdGlvbiBvZiByZXByb2R1Y2libGUgcmVzZWFyY2ggcHJhY3RpY2VzLiANCg0KVXNpbmcgcmVwcm9kdWNpYmxlIHJlc2VhcmNoIHByYWN0aWNlcyAqKnNwZWVkcyB1cCBzY2llbnRpZmljIGRpc2NvdmVyeSoqLCBmb3N0ZXJzICoqZ3JlYXRlciBleGNoYW5nZSBvZiBpZGVhcyBhbW9uZyBzY2llbnRpc3RzKiosIGFuZCAqKnJlZHVjZXMgcmVzZWFyY2ggd2FzdGUqKi5bQFJlZldvcmtzOjIyMzY7QFJlZldvcmtzOjIxODFdIFJlc2VhcmNoIHBhcGVycyB3aXRoIHNoYXJlZCBkYXRhIGhhdmUgKipmZXdlciBlcnJvcnMqKiwgdGhvc2Ugd2l0aCBzaGFyZWQgZGF0YSBvciBzaGFyZWQgY29kZSBhcmUgKipjaXRlZCBtb3JlKiosIGFuZCBzaGFyZWQgZGF0YSBhcmUgKip1c2VkIGluIG1vcmUgcHVibGljYXRpb25zKiouW0BSZWZXb3JrczoyMTgxO0BSZWZXb3JrczoyMzE0XQ0KDQpUaGUgbW9kdWxlcyBpbiB0aGlzIHRvb2xraXQgd2lsbCBndWlkZSB5b3UgdGhyb3VnaCBidWlsZGluZyBhIHJlcHJvZHVjaWJsZSByZXNlYXJjaCBwcm9qZWN0LiBQYXN0cnkgY2hlZnMga25vdyB0aGF0IHRvIHJlLWNyZWF0ZSB0aGF0IG1vc3QgZXhjZWxsZW50IHNvdWZmbGUsIHRoZXkgbXVzdCBoYXZlIHRoZSBzYW1lIGluZ3JlZGllbnRzIGFuZCBhIGNsZWFyIGFuZCBjb21wbGV0ZSByZWNpcGUgdG8gZm9sbG93LiBMaWtld2lzZSwgdG8gcmVwcm9kdWNlIHNjaWVudGlmaWMgcmVzZWFyY2gsIHNjaWVudGlzdHMgbmVlZCBhY2Nlc3MgdG8gZGF0YSBhbmQgYSBjbGVhciBhbmQgY29tcGxldGUgcmVjaXBlIGZvciBkYXRhIG1hbmFnZW1lbnQgYW5kIGFuYWx5c2lzLltAUmVmV29ya3M6MjMwNjtAUmVmV29ya3M6MjMwOTtAUmVmV29ya3M6MjIzNl0gQWZ0ZXIgdGhpcyBicmllZiBpbnRyb2R1Y3RvcnkgbW9kdWxlLCB0aGUgb3RoZXIgZm91ciBtb2R1bGVzIGluIG91ciB0b29sa2l0IHdpbGwgaGVscCB5b3UgbGVhcm46DQoNCiogTW9kdWxlIDI6IFN0cmF0ZWdpZXMgdG8gbWFrZSBwcml2YXRlIGRhdGEgc2hhcmVhYmxlDQoqIE1vZHVsZSAzOiBIb3cgdG8gcHJlcGFyZSBzdGF0aXN0aWNhbCBjb2RlIHRvIHNoYXJlDQoqIE1vZHVsZSA0OiBIb3cgdG8gZG9jdW1lbnQgZGF0YSBhbmQgY29kZSBmb3Igc2hhcmluZw0KKiBNb2R1bGUgNTogSG93IHRvIHNoYXJlIGRhdGEgYW5kIGNvZGUgb24gR2l0SHViDQoNCiMjIEV4YW1wbGVzIG9mIHRvb2xraXQgdXNlIHsjZXhhbXBsZXN9DQoNCldlIGludml0ZWQgc2V2ZXJhbCByZXNlYXJjaGVycyB0byBoZWxwIHRlc3QgYW5kIGltcHJvdmUgdGhlIHRvb2xraXQuIFRoZSByZXNlYXJjaGVycyB1c2VkIHRoZSB0b29sa2l0IHRvIGZvcm1hdCwgb3JnYW5pemUsIGFuZCBwb3N0IG1hdGVyaWFscyBmcm9tIGEgcmVjZW50IHB1YmxpY2F0aW9uLiBUaGUgR2l0SHViIHJlcG9zaXRvcmllcyBmb3IgdGhlc2UgcHJvamVjdHMgYXJlIGF2YWlsYWJsZSBhcyBleGFtcGxlczoNCg0KLSBbRW1lcmdlbmN5IGRlcGFydG1lbnQgdmlzaXRzIHBhcGVyXShodHRwczovL2dpdGh1Yi5jb20vYW50cm9sYXJkL0NvZGluZzJTaGFyZSkNCi0gW01FUFMgcHJvdmlkZXIgbmV0d29yayBwYXBlcl0oaHR0cHM6Ly9naXRodWIuY29tL0VsaW5ldmFuZGVuQnJvZWsvQ29kaW5nMlNoYXJlX01FUFNfUHJvdmlkZXJOZXR3b3JrcykNCi0gW1Bsb3MgT25lIEVORFMgcGFwZXJdKGh0dHBzOi8vZ2l0aHViLmNvbS9zcndlYXZlckdTVS9QTE9TT25lMjAxOEVORFNQYXBlcikNCi0gW0JDIFRyYWluaW5nIHByb2plY3RdKGh0dHBzOi8vZ2l0aHViLmNvbS9nd2JvdW5kcy9CQ19UcmFpbmluZykNCi0gW1NhZmUgU3RvcmFnZSBwcm9qZWN0XShodHRwczovL2dpdGh1Yi5jb20vY2Fzc3BoZC9kYXRhUHJvamVjdCkNCg0KRm9yIGJlc3QgcmVzdWx0cywgd2Ugc3VnZ2VzdCByZXZpZXdpbmcgZWFjaCBtb2R1bGUgaW4gaXRzIGVudGlyZXR5IGJlZm9yZSBpbXBsZW1lbnRpbmcgdGhlIGNoYW5nZXMuIFVzZSBvdXIgW3JlcHJvZHVjaWJpbGl0eSBjaGVja2xpc3RdKGh0dHBzOi8vZ2l0aHViLmNvbS9jb2Rpbmcyc2hhcmUvUmVwcm9kdWNpYmlsaXR5VG9vbGtpdC9ibG9iL21hc3Rlci9PcGVuU2NpRmlkZWxpdHlDaGVja2xpc3QyMDE4LjA1LjMwLnBkZikgdG8ga2VlcCB0cmFjayBvZiB5b3VyIHByb2dyZXNzIGFzIHlvdSBnbyB0aHJvdWdoIHRoZSBtb2R1bGVzLg0KDQpJZiB5b3Ugd2FudCB0byBrbm93IG1vcmUgYWJvdXQgcmVwcm9kdWNpYmxlIHJlc2VhcmNoLCB0aGVyZSBhcmUgbWFueSBvdGhlciBwdWJsaWNhdGlvbnMgW0BSZWZXb3Jrczo3NDtAUmVmV29ya3M6NjA7QFJlZldvcmtzOjc2O0BSZWZXb3JrczoyMjA5O0BSZWZXb3Jrczo2NDtAUmVmV29ya3M6NjU7QFJlZldvcmtzOjY2O0BSZWZXb3Jrczo2NztAUmVmV29ya3M6MjMxMjtAUmVmV29ya3M6MjE4ODtAUmVmV29ya3M6NzBdIGFuZCBvbmxpbmUgW3Jlc291cmNlc10oI3Jlc291cmNlcykgYXZhaWxhYmxlLg0KDQojIyBTb2Z0d2FyZSBhbmQgZGF0YSBmaWxlIGZvcm1hdHMNCg0KV2Uga25vdyB0aGF0IGV2ZXJ5b25lIGhhcyBhIGZhdm9yaXRlLCAqZ28tdG8qIHN0YXRpc3RpY2FsIHNvZnR3YXJlIHBsYXRmb3JtLiBUaHJvdWdob3V0IHRoZSBtb2R1bGVzLCB3ZSBwcm92aWRlIHJlcHJvZHVjaWJsZSBleGFtcGxlcyB1c2luZyBmb3VyIHBsYXRmb3JtcyBwb3B1bGFyIGluIHB1YmxpYyBoZWFsdGg6IFIsIFNBUywgU1BTUywgYW5kIFN0YXRhLiBXZSBob3BlIHRoYXQgdGhpcyB3aWxsIG1ha2Ugb3VyIG1vZHVsZXMgdXNlZnVsIGFuZCBhY2Nlc3NpYmxlIGZvciBldmVyeW9uZS4gDQoNCk9uZSBvdmVyYXJjaGluZyByZWNvbW1lbmRhdGlvbiB3ZSBtYWtlIGlzIHRoZSB1c2Ugb2Ygbm9uLXByb3ByaWV0YXJ5IGZpbGUgZm9ybWF0cyB3aGVuIG1ha2luZyBkYXRhIGFuZCBjb2RlIHB1YmxpYy4gSW4gYWxsIG9yIG1vc3QgcGxhdGZvcm1zLCBkYXRhIGNhbiBiZSBzYXZlZCBvciBleHBvcnRlZCBhcyBhIGNvbW1hLXNlcGFyYXRlZCB2YWx1ZXMgKENTVikgZm9ybWF0LCBhbmQgaW4gbWFueSwgY29kZSBvciBjb2RlYm9va3MgY2FuIGJlIHNhdmVkIGFzIGEgUERGIG9yIG5hdGl2ZSBmb3JtYXQuIElmIHlvdSB1c2UgYSBwbGF0Zm9ybSB3aGVyZSBjb2RlYm9va3MgY2Fubm90IGJlIHNhdmVkIG91dCBpbiBhIG5vbi1wcm9wcmlldGFyeSBmb3JtYXQsIHNlZSB0aGUgZXhhbXBsZXMgaW4gTW9kdWxlIDQ6IEhvdyB0byBkb2N1bWVudCBkYXRhIGFuZCBjb2RlIGZvciBzaGFyaW5nLiBUaGVzZSB0eXBlcyBvZiBmaWxlcyAoKmUuZy4qLCBDU1ZzLCBQREZzLCBUWFRzKSBhbmQgdGhlaXIga2luIGNhbiB0aGVuIGJlIGltcG9ydGVkIGJhY2sgaW50byBtb3N0IG90aGVyIHBsYXRmb3JtcyBzbyB0aGF0IGVuZCB1c2VycyBjYW4gcmVwcm9kdWNlIHdvcmsgaW4gdGhlaXIgcHJlZmVycmVkIHNvZnR3YXJlLiBXZSBoYXZlIHR3byBtYWluIG1vdGl2YXRpb25zIGZvciBwcm9tb3RpbmcgdGhlIHVzZSBvZiBub24tcHJvcHJpZXRhcnkgZm9ybWF0cyBmb3IgcmVwcm9kdWNpYmlsaXR5OiAgMSkgVXNpbmcgbm9uLXByb3ByaWV0YXJ5IGZpbGUgZm9ybWF0cyBpbmNyZWFzZXMgYWNjZXNzaWJpbGl0eSB0byBjb2RlIGFuZCBkYXRhIGFuZCB3aWRlbnMgdGhlIGF1ZGllbmNlIG9mIHBvdGVudGlhbCByZXByb2R1Y2VycyBvZiB5b3VyIGF3ZXNvbWUgd29yazsgYW5kIDIpICpNb25leSouIElmIHlvdSBhcmUgbm8gbG9uZ2VyIGEgc3R1ZGVudCwgdGhpbmsgYmFjayB0byB3aGVuIHlvdSBmb3VuZCBhIHBlcmZlY3QgZGF0YXNldCwgYnV0IGRpZCBub3QgaGF2ZSBhY2Nlc3MgdG8gU3RhdGEgb3IgU0FTLCBhbmQgY291bGRuJ3QgZXZlbiB0aGluayBvZiBidXlpbmcgaXQuIA0KDQojIyBUaGUgY29kaW5nMnNoYXJlIHByb2plY3QNCg0KVGhlIGNvZGluZzJzaGFyZSBwcm9qZWN0IHdhcyBmdW5kZWQgYnkgdGhlIFJvYmVydCBXb29kIEpvaG5zb24gRm91bmRhdGlvbidzIFsiT3BlbiBBY2Nlc3MiIHN0cmF0ZWd5XShodHRwczovL3d3dy5yd2pmLm9yZy9lbi9jdWx0dXJlLW9mLWhlYWx0aC8yMDE2LzA5L2hlbHBfdXNfbWFrZV9yZXNlYXJjLmh0bWwpLiBJbiB0aGUgZmlyc3QgcGFydCBvZiB0aGUgcHJvamVjdCB3ZSBzdXJ2ZXllZCAyMDAgcHVibGljIGhlYWx0aCBzY2llbnRpc3RzIGFib3V0IHRoZWlyIGN1cnJlbnQgcmVwcm9kdWNpYmxlIHJlc2VhcmNoIHByYWN0aWNlcy4gVGhlIGdvYWwgb2YgdGhlIHN1cnZleSB3YXMgdG8gZ2FpbiBhbiB1bmRlcnN0YW5kaW5nIG9mIGN1cnJlbnQgcHJhY3RpY2VzIGFuZCBkZXNpcmVkIHRyYWluaW5nIGZvciByZXByb2R1Y2libGUgcmVzZWFyY2gsIGFuZCB0byB1c2Ugd2hhdCB3ZSBsZWFybmVkIHRvIGRlc2lnbiB0aGVzZSBtb2R1bGVzLiANCg0KT25lIHRoaW5nIHdlIGZvdW5kIHdhcyB0aGF0IGRpc3NlbWluYXRpb24gb2YgZGF0YSBhbmQgY29kZSB3YXMgcmFyZS4gVGhlIG1vc3QgY29tbW9uIHRoaW5nIHJlc2VhcmNoZXJzIHJlcG9ydGVkIHNoYXJpbmcgd2FzIGNsZWFuIGRhdGEsIGFuZCBldmVuIHRoYXQgd2FzIG9ubHkgc2hhcmVkIGJ5IDEyLjQlIG9mIHBhcnRpY2lwYW50cy4gDQoNCmBgYHtyIHdhcm5pbmc9RkFMU0UsIGVjaG89RkFMU0UsIG1lc3NhZ2U9RkFMU0UsIGZpZy5oZWlnaHQgPSAzLCBmaWcud2lkdGggPSA4fQ0KbGlicmFyeShmb3JlaWduKQ0KYWltMSA8LSByZWFkLnNwc3MoIkc6L0NQSFNTL09wZW5TY2llbmNlL0RhdGEvQWltMS9PcGVuU2NpZW5jZUFpbTEuc2F2IiwgdG8uZGF0YS5mcmFtZSA9IFRSVUUpDQoNCmxpYnJhcnkoZ2dwbG90MikNCmxpYnJhcnkoc3RyaW5ncikNCg0KY2xlYW5EYXRhIDwtIHN1bShhaW0xJFEyNV8yXzFbYWltMSRRMjVfMl8xIT0tOTldLCBuYS5ybT1UKS9zdW0oYWltMSRGaW5pc2hlZCwgbmEucm09VCkNCmNsZWFuQ29kZSA8LSBzdW0oYWltMSRRMjVfMl8yW2FpbTEkUTI1XzJfMiE9LTk5XSwgbmEucm09VCkvc3VtKGFpbTEkRmluaXNoZWQsIG5hLnJtPVQpDQpwcm9qRGlyIDwtIHN1bShhaW0xJFEyNV8yXzNbYWltMSRRMjVfMl8zIT0tOTldLCBuYS5ybT1UKS9zdW0oYWltMSRGaW5pc2hlZCwgbmEucm09VCkNCnJlYWRNZSA8LSBzdW0oYWltMSRRMjVfMl80W2FpbTEkUTI1XzJfNCE9LTk5XSwgbmEucm09VCkvc3VtKGFpbTEkRmluaXNoZWQsIG5hLnJtPVQpDQoNCg0KbnVtcyA8LSBjKGNsZWFuRGF0YSwgY2xlYW5Db2RlLCBwcm9qRGlyLCByZWFkTWUpDQpmb3JtYXQgPC0gYygiQSBjbGVhbiB2ZXJzaW9uIG9mIGRhdGEgdXNlZCBmb3IgYW5hbHlzaXMiLCJDbGVhbiBzdGF0aXN0aWNhbCBjb2RlIGZvciByZXN1bHRzIGluY2x1ZGVkIGluIHRoZSBwdWJsaWNhdGlvbiIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAiQSBwcm9qZWN0IGRpcmVjdG9yeSB3aXRoIGRhdGEgYW5kIG9yIHN0YXRpc3RpY2FsIGNvZGUiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgIkEgcmVhZG1lIGZpbGUgZXhwbGFpbmluZyB0aGUgZGF0YSBhbmQgb3Igc3RhdGlzdGljYWwgY29kZSIpDQpjb2RlRm9ybWF0IDwtIGRhdGEuZnJhbWUobnVtcywgZm9ybWF0KQ0KDQoNCmdncGxvdChjb2RlRm9ybWF0LCBhZXMoeD1yZW9yZGVyKGZvcm1hdCwgbnVtcyksIHk9bnVtcykpICsNCiAgICAgICAgIGdlb21fYmFyKHN0YXQ9ImlkZW50aXR5IiwgZmlsbCA9ICIjMDAwMDY2IikgKyANCiAgY29vcmRfZmxpcCgpICsgDQogIHhsYWIoIkRpc3NlbWluYXRpb24gYWZ0ZXIgcHVibGlzaGluZyIpICsgeWxhYigiUGVyY2VudGFnZSBvZiBwYXJ0aWNpcGFudHMiKSArDQogICAgc2NhbGVfeF9kaXNjcmV0ZShsYWJlbHMgPSBmdW5jdGlvbih4KSBzdHJfd3JhcCh4LCB3aWR0aCA9IDMwKSkgKyANCiAgc2NhbGVfeV9jb250aW51b3VzKGxhYmVscyA9IHNjYWxlczo6cGVyY2VudCwgbGltaXRzID0gYygwLCAxKSkgKyANCiAgc2NhbGVfZmlsbF9tYW51YWwoZ3VpZGU9RkFMU0UgKSArDQogIHRoZW1lKHRleHQgPSBlbGVtZW50X3RleHQoc2l6ZT0xNCkpICsNCiAgdGhlbWVfbWluaW1hbCgpDQogIA0KYGBgDQoNCkFib3V0IDI1JSBvZiBwYXJ0aWNpcGFudHMgcmVwb3J0ZWQgX25vdF8gaGF2aW5nIGEgY29kZWJvb2sgZm9yIGEgcmVjZW50IGRhdGEgcHJvamVjdCBhbmQgdGhlIHVzZSBvZiByZWNvbW1lbmRlZCBjb2RpbmcgcHJhY3RpY2VzIHdhcyBub3QgZXZlbiBjbG9zZSB0byB1bml2ZXJzYWw6DQoNCmBgYHtyIHdhcm5pbmc9RkFMU0UsIGVjaG89RkFMU0UsIG1lc3NhZ2U9RkFMU0UsIGZpZy5oZWlnaHQgPSA0LjUsIGZpZy53aWR0aCA9IDh9DQojZGV0ZXJtaW5lIGhvdyBtYW55IGNvZGVycyBpbiBkYXRhIHNldA0KY29kZXJzIDwtIHN1bShhaW0xJEZpbmlzaGVkLCBuYS5ybT1UKSAtIHN1bShhaW0xJFE3XzJfNlthaW0xJFE3XzJfNiE9LTk5XSwgbmEucm09VCkNCg0KbGltaXRlZCA8LSBzdW0oYWltMSRRMjJfMlthaW0xJFEyMl8yIT0tOTldLCBuYS5ybT1UKS9jb2RlcnMNCm1ldGFkYXRhIDwtIHN1bShhaW0xJFEyMl8zW2FpbTEkUTIyXzMhPS05OV0sIG5hLnJtPVQpL2NvZGVycw0Kc2VlZCA8LSBzdW0oYWltMSRRMjJfNFthaW0xJFEyMl80IT0tOTldLCBuYS5ybT1UKS9jb2RlcnMNCmNvbnNpc3RlbnQgPC0gc3VtKGFpbTEkUTIyXzVbYWltMSRRMjJfNSE9LTk5XSwgbmEucm09VCkvY29kZXJzDQp3aGl0ZXNwIDwtIHN1bShhaW0xJFEyMl82W2FpbTEkUTIyXzYhPS05OV0sIG5hLnJtPVQpL2NvZGVycw0KaW5kZW50IDwtIHN1bShhaW0xJFEyMl83W2FpbTEkUTIyXzchPS05OV0sIG5hLnJtPVQpL2NvZGVycw0KZnVuY3QgPC0gc3VtKGFpbTEkUTIyXzhbYWltMSRRMjJfOCE9LTk5XSwgbmEucm09VCkvY29kZXJzDQpyZXN1bHRzIDwtIHN1bShhaW0xJFEyMl85W2FpbTEkUTIyXzkhPS05OV0sIG5hLnJtPVQpL2NvZGVycw0KbGl0ZXJhdGUgPC0gc3VtKGFpbTEkUTIyXzEwW2FpbTEkUTIyXzEwIT0tOTldLCBuYS5ybT1UKS9jb2RlcnMNCg0KbnVtcyA8LSBjKGxpbWl0ZWQsbWV0YWRhdGEsc2VlZCxjb25zaXN0ZW50LHdoaXRlc3AsaW5kZW50LGZ1bmN0LA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJlc3VsdHMsbGl0ZXJhdGUpDQpmb3JtYXQgPC0gYygiTGltaXRlZCBsaW5lcyBvZiBjb2RlIHRvIGEgY2VydGFpbiBsZW5ndGgiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgIkluY2x1ZGVkIG1ldGFkYXRhLCBzdWNoIGFzIHRoZSBkYXRlIG9yIHByb2plY3QgbmFtZSwgaW4gdGhlIGZpbGUgdGl0bGUiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgIkluY2x1ZGVkIHNlZWQgdmFsdWVzIGZvciBhbmFseXNlcyB0aGF0IGluY2x1ZGVkIHJhbmRvbW5lc3MiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgIlVzZWQgYSBjb25zaXN0ZW50IHdheSB0byBuYW1lIHZhcmlhYmxlcyBhbmQgZnVuY3Rpb25zIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICJTZXBhcmF0ZWQgYW5hbHlzaXMgc3RlcHMgd2l0aCB3aGl0ZSBzcGFjZSBvciBibGFuayBsaW5lcyIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAiVXNlZCBpbmRlbnRhdGlvbiB0byBncm91cCBsaW5lcyBvZiBjb2RlIHdpdGhpbiBwcm9jZWR1cmVzL2Z1bmN0aW9ucyIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAiV3JvdGUgZnVuY3Rpb25zIGZvciB0YXNrcyByZXBlYXRlZCBtdWx0aXBsZSB0aW1lcyIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAiSW5jbHVkZWQgc29tZSByZXN1bHRzIHdpdGhpbiB0aGUgYW5ub3RhdGlvbiIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAiVXNlZCBsaXRlcmF0ZSBwcm9ncmFtbWluZyBwcm9jZWR1cmVzIHRoYXQgaW50ZWdyYXRlIGNvZGUgd2l0aCB0ZXh0IGFuZCByZXN1bHRzIikNCmNvZGVGb3JtYXQgPC0gZGF0YS5mcmFtZShudW1zLCBmb3JtYXQpDQoNCmdncGxvdChjb2RlRm9ybWF0LCBhZXMoeD1yZW9yZGVyKGZvcm1hdCwgbnVtcyksIHk9bnVtcykpICsNCiAgICAgICAgIGdlb21fYmFyKHN0YXQ9ImlkZW50aXR5IiwgZmlsbCA9ICIjMDAwMDY2IikgKw0KICBjb29yZF9mbGlwKCkgKw0KICB4bGFiKCJCYXJyaWVycyB0byByZXByb2R1Y2libGUgcmVzZWFyY2giKSArIHlsYWIoIlBlcmNlbnRhZ2Ugb2YgcGFydGljaXBhbnRzIikgKw0KICAgIHNjYWxlX3hfZGlzY3JldGUobGFiZWxzID0gZnVuY3Rpb24oeCkgc3RyX3dyYXAoeCwgd2lkdGggPSAzMCkpICsgDQogIHNjYWxlX3lfY29udGludW91cyhsYWJlbHMgPSBzY2FsZXM6OnBlcmNlbnQsIGxpbWl0cyA9IGMoMCwgMSkpICsgDQogIHNjYWxlX2ZpbGxfbWFudWFsKGd1aWRlPUZBTFNFICkgKw0KICB0aGVtZSh0ZXh0ID0gZWxlbWVudF90ZXh0KHNpemU9MTQpKSArIA0KICB0aGVtZV9taW5pbWFsKCkNCmBgYA0KDQoNClN1cnZleSBwYXJ0aWNpcGFudHMgcmVwb3J0ZWQgdGhhdCBkYXRhIHByaXZhY3kgd2FzIHRoZSBtb3N0IGNvbW1vbiBiYXJyaWVyIHRvIG1ha2luZyBkYXRhIG9yIHN0YXRpc3RpY2FsIGNvZGUgcHVibGljbHkgYXZhaWxhYmxlLg0KDQpgYGB7ciB3YXJuaW5nPUZBTFNFLCBlY2hvPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCBmaWcuaGVpZ2h0ID0gMywgZmlnLndpZHRoID0gOH0NCg0KdGltZSA8LSBzdW0oYWltMSRRMzJfMSwgbmEucm09VCkvY29kZXJzDQprbm93bGVkZ2UgPC0gc3VtKGFpbTEkUTMyXzIsIG5hLnJtPVQpL2NvZGVycw0KcHJpdmFjeSA8LSBzdW0oYWltMSRRMzJfMywgbmEucm09VCkvY29kZXJzDQpwcm9wZXJ0eSA8LSBzdW0oYWltMSRRMzJfNCwgbmEucm09VCkvY29kZXJzDQpjb21wZXRpdGlvbiA8LSBzdW0oYWltMSRRMzJfNSwgbmEucm09VCkvY29kZXJzDQplcnJvcnMgPC0gc3VtKGFpbTEkUTMyXzYsIG5hLnJtPVQpL2NvZGVycw0KaW5jZW50aXZlIDwtIHN1bShhaW0xJFEzMl83LCBuYS5ybT1UKS9jb2RlcnMNCg0KbnVtcyA8LSBjKHRpbWUsa25vd2xlZGdlLHByaXZhY3kscHJvcGVydHksY29tcGV0aXRpb24sZXJyb3JzLGluY2VudGl2ZSkNCmZvcm1hdCA8LSBjKCJMYWNrIG9mIHRpbWUiLCJMYWNrIG9mIGtub3dsZWRnZS90cmFpbmluZyBvbiByZXByb2R1Y2libGUgcmVzZWFyY2ggcHJhY3RpY2VzIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICJEYXRhIHByaXZhY3kiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgIkludGVsbGVjdHVhbCBwcm9wZXJ0eSBjb25jZXJucyIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAiUHJvZmVzc2lvbmFsIGNvbXBldGl0aW9uIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICJDb25jZXJucyBvZiBlcnJvcnMgYmVpbmcgZGlzY292ZXJlZCIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAiTGFjayBvZiBpbmNlbnRpdmUiKQ0KY29kZUZvcm1hdCA8LSBkYXRhLmZyYW1lKG51bXMsIGZvcm1hdCkNCg0KDQpnZ3Bsb3QoY29kZUZvcm1hdCwgYWVzKHg9cmVvcmRlcihmb3JtYXQsIG51bXMpLCB5PW51bXMpKSArDQogICAgICAgICBnZW9tX2JhcihzdGF0PSJpZGVudGl0eSIsIGZpbGwgPSAiIzAwMDA2NiIpICsNCiAgY29vcmRfZmxpcCgpICsNCiAgeGxhYigiQmFycmllcnMgdG8gcmVwcm9kdWNpYmxlIHJlc2VhcmNoIikgKyB5bGFiKCJQZXJjZW50YWdlIG9mIHBhcnRpY2lwYW50cyIpICsNCiAgICBzY2FsZV94X2Rpc2NyZXRlKGxhYmVscyA9IGZ1bmN0aW9uKHgpIHN0cl93cmFwKHgsIHdpZHRoID0gMzApKSArIA0KICBzY2FsZV95X2NvbnRpbnVvdXMobGFiZWxzID0gc2NhbGVzOjpwZXJjZW50LCBsaW1pdHMgPSBjKDAsIDEpKSArIA0KICBzY2FsZV9maWxsX21hbnVhbChndWlkZT1GQUxTRSApICsNCiAgdGhlbWUodGV4dCA9IGVsZW1lbnRfdGV4dChzaXplPTE0KSkgKyANCiAgdGhlbWVfbWluaW1hbCgpDQpgYGANCg0KU3VydmV5IHBhcnRpY2lwYW50cyBhbHNvIHRob3VnaHQgdGhhdCB0cmFpbmluZywgcmVxdWlyZW1lbnRzLCBhbmQgaW5jZW50aXZlcyB3b3VsZCBmYWNpbGl0YXRlIHVzZSBvZiBtb3JlIHJlcHJvZHVjaWJsZSByZXNlYXJjaCBwcmFjdGljZXM6ICANCg0KYGBge3Igd2FybmluZz1GQUxTRSwgZWNobz1GQUxTRSwgbWVzc2FnZT1GQUxTRSwgZmlnLmhlaWdodCA9IDMsIGZpZy53aWR0aCA9IDh9DQpjb2RlcnMgPC0gc3VtKGFpbTEkRmluaXNoZWQsIG5hLnJtPVQpIC0gc3VtKGFpbTEkUTdfMl82W2FpbTEkUTdfMl82IT0tOTldLCBuYS5ybT1UKQ0KaHVtYW5zIDwtIHN1bShhaW0xJFEzMV8xW2FpbTEkUTMxXzEhPS05OV0sIG5hLnJtPVQpL2NvZGVycw0KdHJhaW5pbmcgPC0gc3VtKGFpbTEkUTMxXzJbYWltMSRRMzFfMiE9LTk5XSwgbmEucm09VCkvY29kZXJzDQpmdW5kZXIgPC0gc3VtKGFpbTEkUTMxXzNbYWltMSRRMzFfMyE9LTk5XSwgbmEucm09VCkvY29kZXJzDQpqb3VybmFsIDwtIHN1bShhaW0xJFEzMV80W2FpbTEkUTMxXzQhPS05OV0sIG5hLnJtPVQpL2NvZGVycw0KaW5jZW50aXZlcyA8LSBzdW0oYWltMSRRMzFfNVthaW0xJFEzMV81IT0tOTldLCBuYS5ybT1UKS9jb2RlcnMNCg0KbnVtcyA8LSBjKGh1bWFucywgdHJhaW5pbmcsZnVuZGVyLGpvdXJuYWwsaW5jZW50aXZlcykNCmZvcm1hdCA8LSBjKCJBZGRpdGlvbmFsIGh1bWFuIGFuZCBmaW5hbmNpYWwgcmVzb3VyY2VzIiwiVHJhaW5pbmcgb24gcmVwcm9kdWNpYmxlIHJlc2VhcmNoIHByYWN0aWNlcyIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAiUmVxdWlyZW1lbnRzIGJ5IGZ1bmRlcnMgdG8gZGlzc2VtaW5hdGUgZGF0YSBhbmQgc3RhdGlzdGljYWwgY29kZSIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAiUmVxdWlyZW1lbnRzIGJ5IGpvdXJuYWxzIHRvIGluY2x1ZGUgYWNjZXNzIHRvIGRhdGEgYW5kIHN0YXRpc3RpY2FsIGNvZGUiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgIldvcmtwbGFjZSBpbmNlbnRpdmVzIChlLmcuLCBwYXkgaW5jcmVhc2VzIG9yIGNyZWRpdCB0b3dhcmQgdGVudXJlKSIpDQpjb2RlRm9ybWF0IDwtIGRhdGEuZnJhbWUobnVtcywgZm9ybWF0KQ0KDQoNCmdncGxvdChjb2RlRm9ybWF0LCBhZXMoeD1yZW9yZGVyKGZvcm1hdCwgbnVtcyksIHk9bnVtcykpICsNCiAgICAgICAgIGdlb21fYmFyKHN0YXQ9ImlkZW50aXR5IiwgZmlsbCA9ICIjMDAwMDY2IikgKw0KICBjb29yZF9mbGlwKCkgKw0KICB4bGFiKCJGYWNpbGl0YXRvcnMgb2YgcmVwcm9kdWNpYmxlIHJlc2VhcmNoIikgKyB5bGFiKCJQZXJjZW50YWdlIG9mIHBhcnRpY2lwYW50cyIpICsNCiAgICBzY2FsZV94X2Rpc2NyZXRlKGxhYmVscyA9IGZ1bmN0aW9uKHgpIHN0cl93cmFwKHgsIHdpZHRoID0gMzApKSArIA0KICBzY2FsZV95X2NvbnRpbnVvdXMobGFiZWxzID0gc2NhbGVzOjpwZXJjZW50LCBsaW1pdHMgPSBjKDAsIDEpKSArIA0KICBzY2FsZV9maWxsX21hbnVhbChndWlkZT1GQUxTRSApICsNCiAgdGhlbWUodGV4dCA9IGVsZW1lbnRfdGV4dChzaXplPTE0KSkgKyANCiAgdGhlbWVfbWluaW1hbCgpDQoNCmBgYA0KDQpNb3JlIGRldGFpbGVkIHJlc3VsdHMsIGRhdGEsIGFuZCBzdGF0aXN0aWNhbCBjb2RlIHdpbGwgYmUgbWFkZSBhdmFpbGFibGUgaW4gb3VyIFtHaXRIdWIgcmVwb3NpdG9yeV0oaHR0cHM6Ly9naXRodWIuY29tL2NvZGluZzJzaGFyZSkgc2hvcnRseS4NCg0KIyMgUmVzb3VyY2VzIHsjcmVzb3VyY2VzfQ0KDQojIyMgT25saW5lIHJlc291cmNlcw0KLSBbUmVwcm9kdWNpYmlsaXR5IC0gV2lraXBlZGlhXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9SZXByb2R1Y2liaWxpdHkpDQotIFtSZXByb2R1Y2liaWxpdHkgR3VpZGUgLSBUaGUgck9wZW5TY2kgUHJvamVjdF0oaHR0cDovL3JvcGVuc2NpLmdpdGh1Yi5pby9yZXByb2R1Y2liaWxpdHktZ3VpZGUvc2VjdGlvbnMvaW50cm9kdWN0aW9uLykNCi0gW1RoZSBSZWFsIFJlYXNvbiBSZXByb2R1Y2libGUgUmVzZWFyY2ggaXMgSW1wb3J0YW50XShodHRwczovL3NpbXBseXN0YXRpc3RpY3Mub3JnLzIwMTQvMDYvMDYvdGhlLXJlYWwtcmVhc29uLXJlcHJvZHVjaWJsZS1yZXNlYXJjaC1pcy1pbXBvcnRhbnQvKQ0KLSBbQ1JBTiBUYXNrIFZpZXc6IFJlcHJvZHVjaWJsZSBSZXNlYXJjaF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3ZpZXdzL1JlcHJvZHVjaWJsZVJlc2VhcmNoLmh0bWwpDQotIFtSZXRyYWN0aW9uIFdhdGNoXShodHRwczovL3JldHJhY3Rpb253YXRjaC5jb20vKQ0KLSBbUk9wZW5TY2ldKGh0dHBzOi8vcm9wZW5zY2kub3JnLykNCi0gW2NvZGluZzJzaGFyZTogVHdpdHRlcl0oaHR0cHM6Ly90d2l0dGVyLmNvbS9jb2Rpbmcyc2hhcmU/bGFuZz1lbikNCi0gW2NvZGluZzJzaGFyZTogR2l0SHViXShodHRwczovL2dpdGh1Yi5jb20vY29kaW5nMnNoYXJlKQ0KDQoNCiMjIyBSZWZlcmVuY2VzDQo=


Reproducibility Toolkit on GitHub

Top