audit control environment
DESCRIPTION
Audit Control Environment. Mike Smorul UMIACS. Issues surrounding asserting integrity. Threats to Integrity of Digital Archives Hardware/media degradation Hardware/software malfunction Operational errors Security breaches, malicious alterations Technology evolution - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/1.jpg)
Audit Control Audit Control EnvironmentEnvironment
Mike SmorulUMIACS
![Page 2: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/2.jpg)
Issues surrounding asserting Issues surrounding asserting integrityintegrity
• Threats to Integrity of Digital Archives– Hardware/media degradation– Hardware/software malfunction– Operational errors– Security breaches, malicious alterations– Technology evolution – Object transformation (format obsolescence)– Infrequent access to most data
![Page 3: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/3.jpg)
Using Hashes to Monitor Using Hashes to Monitor FilesFiles
• Strong hashes can assert a file has not changed
• How to manage millions of hashes?• How do you prove the hash value hasn’t
changed?• How do you prove a hash value was issued
at a given time?
![Page 4: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/4.jpg)
Audit Control Environment Audit Control Environment (ACE)(ACE)
• Solves the problem of storing and verifying hashes.
• Secures hashes by issuing token for each file/hash to me monitored.
• Tokens contain a cryptographic proof that allows for 3rd party auditing.
• One number stored externally is used to audit tokens and hashes.
![Page 5: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/5.jpg)
Hash AuthenticationHash Authentication
Hash 1 Hash 6Hash 5Hash 3 Hash 4Hash 2
Intermediate Hash ValuePrevious Round Hash
CSI (one hash value)
ChallengeHash
IHV
IHV
Gather Hashes During RoundCreate Merkel Tree For Supplied HashesLink to previous roundGenerate proof for hash
![Page 6: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/6.jpg)
Token SampleToken Sample<token>
<token-class>SHA-256-0</token-class><digest-service>SHA-256</digest-service><name>/SRB3_2_1f.tar</name><round-id>1223</round-id><time-stamp>2008-07-22T11:03:45.059-0400</time-stamp><proof><element index="0"><hash>2e869e2ce41ede3ceb3af50f8aa2705067b3e67055b5b3d2787e2c294a95a869</hash></element><element index="0"><hash>6a925501991d7b4ff660d499416fd45a20dde161eb68e59fedc0f58208ad86cf</hash></element><element index="0"><hash>134432a6a6527162d24e99435e817511eeb89ddc03afbc6a30f23e404847cc06</hash></element><element index="1"><hash>1aeaf2d76976cf9759b0d63bc7acdf9c6df68875bfc9bcc0e22c19401aab0133</hash></element></proof>
</token>
![Page 7: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/7.jpg)
How to scale?How to scale?• Two layers of Merkel tree
1. Short rounds(seconds), that generate Cryptographic Summary Information(CSI).
2. Each successive round includes previous CSI3. Second, daily rounds comprised of all CSI’s for
previous day.• Daily tree root, called Witness can validate all
CSI’s for a day.– Only 365/year generated. Very manageable!
• Two components, an Integrity Management Service(IMS), and Audit Manager(AM) were developed.
![Page 8: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/8.jpg)
ComponentsComponents• Integrity Management Service (remote)
– Runs all hash aggregation, round generation, witness publication.
– Stores CSI values– Generate proofs from CSI to witness– ims.umiacs.umd.edu
• Audit Manager (local)– Monitors local files– Determines audit policy– One or more per archive– Locally stores hashes and tokens
![Page 9: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/9.jpg)
ACE – System ArchitectureACE – System Architecture
reply
Token Registry
hdd
Archiving Node
cd-romtape drive
request
ACE Audit Manager
Third-Party Integrity Management System
CryptoSummary
Information
reply
Token Registry
hdd
Archiving Node
cd-romtape drive
request
ACE Audit Manager
witnesses witnesses
Audit Policy Audit Policy
![Page 10: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/10.jpg)
ACE AuditACE Audit Audit Local Files: Audit Manager periodically scans all
files and compares stored digests with computed digests. Assume valid hashes in database
Audit Local Manager: Manager computes round summary for each digest using that digest and its token. This is compared to value stored on the IMS. Assume IMS returns valid summary information, do not trust
hashes in database
IMS Audit: Round summaries are used to compute witness values. These are compared with offsite witness values. Do not trust IMS, force IMS to prove its CSIs link to a witness
![Page 11: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/11.jpg)
Audit ManagerAudit Manager• Downloadable, one or more per archive• Monitors local files• Simple Requirements
– Java 1.6+– Tomcat– MySQL
• Managed by archivist/librarian after install• Monitor multiple collections on different
architectures• Hides all the complexity you just saw!
![Page 12: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/12.jpg)
ACE Audit ManagerACE Audit Manager• Dashboard for collection monitoring
![Page 13: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/13.jpg)
LoggingLogging• All events logged
– Files offline, corrupt, re-available– Audit times, last seen
![Page 14: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/14.jpg)
Error ReportingError Reporting• View all files that are not perfectly intact• Remove files from monitoring.• View events associated with files
![Page 15: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/15.jpg)
Audit Manager FeaturesAudit Manager Features• Compare collections to hash list
– Same/different names, same/differing digests• Export collection details
– Hash export, wget crawl export• JSON interface for embedding statistics in
3rd party websites• Gateway to data
![Page 16: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/16.jpg)
Fun with hashesFun with hashes• Ensure everything was uploaded
– Accepts a bag-it style manifest (hash + path)• Extract all the duplicates
– More exist than you think• Ensure collection replicas are complete• Locate renamed files
![Page 17: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/17.jpg)
PerformancePerformance• Audit Manager (1.1beta3)• 1.25 million false digests (no bytes read)
– Registration: 3h, 6m (112 files/s)– Audit: 1h, 15m (277 files/s)
• 1.25 million false data files (1.25Tb data)– Registration: 5h, 7m (67.8 files/s, 67.8MBytes/s)– Audit: 4h, 30m (77.2 files/s, 77.2MBytes/s)
• In practice, bottleneck tends to occur at archival resource, not AM.
![Page 18: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/18.jpg)
Chronopolis testChronopolis test• Three sites
– UMD, NCAR, SDSC• Three ACE AM installations• Independent monitoring at all sites• 30 day audit policy• Over 17Tb and 5.5 million files
![Page 19: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/19.jpg)
Chronopolis TestChronopolis TestProvider Files Size(GB
)Time(h) Files/s MB/s
CDL 46,762 4,291 20:32 .63 59.44
SIO-GDC
197,718 815 6:49 8.05 34.00
ICPSR 4,830,625
6,957 122:48 10.93 16.11
NC-State
608,424 5,465 32:14 5.24 48.22
![Page 20: Audit Control Environment](https://reader036.vdocuments.net/reader036/viewer/2022081604/5681522a550346895dc07299/html5/thumbnails/20.jpg)
ACE SummaryACE Summary• High performance, Scalable• 3rd party auditable• Version 1.0 publically available
– Support for local files, SRB, iRODS
• http://adapt.umiacs.umd.edu/ace
• Standalone client available