a short introduction to multimedia forensics the science discovering the history of multimedia...
TRANSCRIPT
A short introduction to Multimedia
Forensics: the science
discovering the history of
multimedia contents
Sebastiano BattiatoDipartimento di Matematica e Informatica,
Università di Catania
Image Processing LAB – http://iplab.dmi.unict.it
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Forensic science (often shortened to forensics) is the practical
application of science to matters of the law. Use of scientific
methods for gaining probative facts (from physical/analog or
digital evidences )
History of Digital Forensics• Earliest notion of digital forensics came when the Federal
Rules (US) of evidence first started to discuss digitalevidence in the 1970s
• Real digital forensics investigations started in the mid-to late1980s when federal agents had to start figuring out ways tosearch computers for digital evidence
• This “home-grown”, bottom-up approach continued until thelate 1990s when security researchers at universities andlabs started to figure out that this problem was big enough towarrant investigation.
• First research groups started around 2000 or 2001.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
The first Digital Forensics Research
Workshop (DFRWS) was held in Utica, NY in
August 2001.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia
Forensics
- Source identification
- Integrity verification/tampering detection
Techniques from multimedia forensics merely provide a way to
test for the authenticity and source of digital sensor data. In this
sense is not about analyzing the semantics of digital or
digitized media objects.
Multimedia Forensics (in academic
literature)
• The terms appeared in early 2000
• K. J. Ray Liu, on “Multimedia Forensics: Where
Sherlock Holmes Meets Signal Processing” Invited
talk @ ICME 2006
• Multimedia Forensics is not Computer Forensics
(2009, Bohem et al.)
• Recent Surveys on the field (Stamm 2013, Piva 2013)
• WIFS, TIFS, Dedicated workshop at major conference
(ACM MM, ICME, ICIAP), Special issues on Int. Journal
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Forensics (in practice)
• Source Identification
• Integrity/Authenticity
• Enhancement/Restoration
• Interpretation and Content Analysis– Plate Recognition
– Dynamic Reconstruction (car crashes, etc.)
– Antropomethric issues
– …
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
“Forensics Image (Video) analysis is
the application of IMAGE SCIENCE
and DOMAIN EXPERTISE to interpret
the content of an image or the image
itself in legal matters” (SWGIT –
www.fbi.gov)
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Recent documents:
• 2015-09-29 SWGDE Best Practices for Photographic Comparison for AllDisciplines
• 2015-09-29 SWGDE Image Processing Guidelines
• 2015-09-29 SWGDE Proposed Techniques for Advanced Data Recoveryfrom Security Digital Video Recorders v1-1
• 2015-09-29 SWGDE Training Guidelines for Video Analysis, ImageAnalysis and Photography V1-1
https://www.swgde.org/
Fantasy/Fiction
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
CSI
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Reality
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
I Need That Plate! No Way...
Fantasy
• We cannot introduce new information
(there is no data..)
• But we can enhance and extract only
something already present
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
• What is the minimum quality for video? “Minimum quality”
doesn’t exist.
• The success of the enhancement depends on several
factors:
Main goal (video captured with an HD camera but the
license plate we need to extract is too far away)
Technical related details: Resolution of the area of
interest, Level of compression, Presence of blur /
focus, Number of available frames, Noise / brightness
and contrast
• It’s important to understand which defects are
present in order to apply the proper tools.
Understand When It’s Possible To Get
Something
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Perspective Correction
Geometrical transformation
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Forensics is based on the idea
that inherent traces (like digital fingerprints)
are left behind in a digital media during both
the creation phase and any other
successively process.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
• Example:• Forensic analysis of a smartphone: which pictures have been generated
on the device and which ones have been generated by other devices
and sent by messaging application or saved from the internet
• We can identify:• Type of device
• Maker and model
• Specific exemplar
Camera BallisticsWhich Device Has Created This Picture?
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Source Identification Noise Based
Sensor output carries not only pure signal
but also various noise components. Sensor
noise model could be used as a
representative feature for cameras.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Sensor Identification Using
Pattern Noise
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Sensor Identification Using
Pattern NoiseThis method provide good results, and is
quite reliable also using:
–images with different level of JPEG
compression (low, medium and high)
–images processed using point-wise operator
such as brightness/contrast adjustment or
gamma correction.
–images acquired by two cameras of the same
brand and model.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Integrity: What is a Forgery?• “Forgery” is a
subjective word.
• An image canbecome a forgery
based upon the
context in which
it is used.
• An image altered for fun or someone who has taken an badphoto, but has been altered to improve its appearancecannot be considered a forgery even though it has beenaltered from its original capture.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
What is a Forgery?
• The other side of forgery are those who perpetuate aforgery for gain and prestige
• They create an image in which to dupe the recipient intobelieving the image is real and from this be able to gainpayment and fame
• Three type of forgery can be identified:•An image that is created using graphical software•An image where the content has been altered•An image where the context has been altered
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
The content has been alteredDuping the recipient into believing that the objects in animage are something else from what they really are!
November 1997: After 58 tourists were killed in a terrorist attack at thetemple of Hatshepsut in Luxor Egypt, the Swiss tabloid Blick digitally altereda puddle of water to appear as blood flowing from the temple.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
The context has been altered
• Objects are be removed or added,for example, a person can beadded or removed
• The easiest way is to cut anobject from one image and insertit into another image – imageediting software makes this asimple task
• An example is this altered imagewhich could be used to influenceevents in foreign countries whichare not aware of manipulation.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Altering Images
The concepts have movedinto the digital world byvirtue of digital camerasand the availability ofdigital image editingsoftware
The ease of use of digital image editing software, which doesnot require any special skills, makes image manipulation easyto achieve.
circa 1860: This nearly iconic portrait of U.S. President Abraham Lincoln is a composite of Lincoln's head and the Southern politician John Calhoun's body.
27
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Off-side (February 2011)
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Barcelona – Athletico Bilbao
Sydney conman
who photoshopped
pictures of himself
with famous people
gets 12 years’jailhttp://www.smh.com.au
31
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
More (and more) examples
Photo Tampering through History
http://www.fourandsix.com/photo-tampering-history/
Photoshopdisaster
http://www.photoshopdisasters.com/
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
http://petapixel.com/2013/05/08/how-photographers-photoshopped-their-pictures-back-in-1946/Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Image Editing (2)
Malicious image editing alters the image semantic
content, mainly:
Adding information
Removing information
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Piva 2013
Image Editing (2)• Splicing (two images)
– Also called cut and paste, compositing
– Used to add information
• Cloning (single image)
– Also called copy and paste, copy move, region duplication
– Used to add or remove information
– Can be exact, or the clone can be resized, rotated…)
• Inpainting (kind of intelligent clone)
– Seam carving, content aware resize, content aware fill, content dependent crop
– Used to remove information
• Retouch (local editing)
– Dodge and burn, healing tool…
• Image enhancement/filtering
– Histogram equalization, contrast enhancement, median filtering, denoise, smooth…
• Image editing (geometric transformation)
– Resize, crop, zoom, shear
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Analog Vs. Digital• Analogue evidence (audio, photo, video)
– Has always an original (negative or positive)
– Copies are never an exact replica of the original
– Copies of copies will be further deteriorated
• Digital evidence
– Only a finite sequence of number
– Exact copies can be made
– No loss of information between generations
– Every copy of a digital data can be considered the original,
unless is tagged in order to be distinguished by copies
Best evidence clause (Federal Rules of Evidence 1003): if data is stored
in a computer or similar device, any printout readable by sight, shown to
reflect the data accurately, is an “original”.36
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Original File: Special Cases
• Recapture: create a fake and then take a
picture with the camera we want to
pretend the picture was taken with
• Staging: the image file is authentic, but
the content has been staged
In these cases an authentic file does not
imply an authentic content.37
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
How To Authenticate An Image?
• Visual Inspection
• File AnalysisFile Format and Structures
Metadata (EXIF)
Compression Parameters (Quantization
Tables)
• Global AnalysisPixel and compressed data statistics
• Local AnalysisFinding inconsistencies of pixel statistics
across the image
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Image Forensics Methods
Active Methods: Hiding a mark or amessage in a picture when it is created:
WATERMARKINGDrawbacks:• limitate to specially equipped
(=expensive) digital cameras;• not so robust.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Image Forensics Methods
Passive Methods: Using the alterations ofthe underlying statistics produced by digitalforgeries on an image:
PHYSICS BASEDCAMERA BASED
PIXEL BASEDGEOMETRIC BASED
FORMAT BASEDMultimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
INTERPOLATION
LENS CFA SENSOR
POST PROCESSINGDIGITAL IMAGESTORAGE
Processing and Storage
ORIGINALIMAGE
Acquisition
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Types Of Analysis: Scene Level• Use features of the scene, not of the image pixels and could work well also
on scanned photos
• Difficult to spoof
– Most forgeries contains slight errors not visible by human eye but detectable by
proper analysis
• Difficult to automatize the analysis
– Requires a lot of experience
• Better results on splicing, less successful on forgeries which are hiding
something
• Physics based:
– Lighting
• Geometry and perspective inconsistencies
– Principal point analysis
– Shadows
– Photogrammetry
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Types Of Analysis: Signal Level
Based on statistical features of pixel values; need good quality image
• Clone detection
– Cloned image blocks
– Similar couples of key points
• Resampling detection
– For resize, rotate, but also when splicing or cloning
• Enhancement Detection
– Specific for algorithms (median, histogram equalization, color adjustment)
• Seam carving detection
• General intrinsic footprints
• Inconsistencies from acquisition and coding fingerprints
– CFA, PRNU, DCT, ELA…
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Format-BasedJPEG compression engine
(for both luminance and chrominance channels):
the input image ispartitioned into 8x8non-overlapping blocks
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
A JPEG image (luminance component only)
Format-Based
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Format-Based
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Format-BasedJPEG compression engine
(for both luminance and chrominance channels):
the input imageis partitionedinto 8x8 non-overlappingblocks
a DCT transformis applied to eachblock
Format-BasedJPEG forgery engine
THE TYPICAL PIPELINEFOR A COPY-PASTE
OPERATION
+
=
original image
QF(1) = q1
resulting image
QF(3) = q3
2nd image
QF(2) = q2
duplicating
resizing
Order of quantization
Top row: single quantized with steps 2 (left) and 3 (right).
Bottom row: double quantized with steps 3 followed by 2 (left), and 2
followed by 3 (right)
F. Galvan, G. Puglisi, A. R. Bruna, S. Battiato, First Quantization Matrix Estimation from Double Compressed JPEG Images, IEEE Transactions on Information Forensics and Security, 2014.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Current Trends And
Challenges
Current Trends: Point&Shoot
and Share…
Social MF on Facebook
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Preliminar studies involving:
- Different devices
- Dataset of images:
- Different Scene (outdoor artificial,
outdoor natural, indoor)
- Different Quality (resolution and
compression)
- Different upload setting
has proven that some invariance could be
guaranteed.
Some specific editing could be then traced
and used to retrieve useful info about
image before uploading.
Moltisanti, Paratore, Battiato, Saravo - Image Manipulation on Facebook for Forensics
Evidence – ICIAP 2015, LNCS 2015;
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Social (Multimedia) Forensics
• Image and Video Phylogeny
ReVeal project
Video Fingerprint
• Photo DNA
(http://www.microsoftphotodna.com/ )
• VideoGenome
(http://v-nome.org/ )
• Videntifier
(http://www.eff2.net/ )
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
73
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
PRNU on Video (online)Each video is re-encoded (compressed) after it is uploaded. This
compression is needed to reduce the bandwidth usage by reducing the file
size and thus increasing streaming speeds.
By re-encoding the video, the video is altered which makes it harder to
identify the source.
Related performances depends on the brand of the camera, the resolution
and the amount of compression.
The reliability of the PRNU pattern decreases exponentially if the
compression becomes higher.
Challenges:
- Video obtained by some post processing (e.g., transition effects, fusion of
more than one source, …)
- No reference available
- Various Encoders
Who Cares?
media
Who Cares?
geopolitics…
…and political propaganda
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Advertisement
WorldPhotoPress 2012: faked?
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
When is an image fake, and when
is it merely enhanced?
WPP Report:The integrity of the
Image (Nov. 2014)
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Current practices and accepted
standards relating to the
manipulation of still images in
photojournalism and documentary
photography.
Media organizations
prohibit the alteration of
images beyond
traditional darkroom
techniques.
It means – as first – that the
alteration of images – where
alteration means the digital
addition or subtraction of
elements is forbidden
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Data Manipulation on Science: Images
Gilbert, N. “Science Journals Crack down on Image Manipulation.” Nature (2009):
doi:10.1038/news.2009.991.
Source: Office For Research Integrity (ORI)
Forgery on Biomedical Images
Corriere della Sera – Ottobre 2013
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Plagiarized paper:Fabrication of Monodisperse Magnetic Fe3O4-SiO2 Nanocomposites with Core-Shell Structures Hua Fang,* Chun-yang Ma, Tai-li Wan, Mei Zhang, and Wei-hai Shi J. Phys. Chem C 2007, 111, 1065-1070
Original PaperOriented Assembly of Fe3O4 Nanoparticles into Monodisperse Hollow Single-Crystal Microspheres Yu et al, J. Phys. Chem. B 2006, 110, 21667-21671 (Figure 3)
M. Rossner and K. M. Yamada, “What’s in a picture? The temptation of image manipulation.,”
J. Cell Biol., vol. 166, no. 1, pp. 11–5, Jul. 2004.
Forgery on Science
Forgery on Science
“What’s in a picture? The temptation of image manipulation.,” J. Cell Biol., vol. 166, no. 1, pp.
11–5, Jul. 2004.
Body-Worn camera
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Image Manipulation: Case “Mozzarella Blu”
Multimedia Truthfulness
Verification in Legal
Environment and Social
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness
Verification in Legal
Environment and Social
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness
Verification in Legal
Environment and Social
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Evidence on the web
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
http://revealproject.eu/
http://www.rewindproject.eu/
http://maven-project.eu/#_=_
https://s-five.eu/
The first public draft of the FIVE Best
Practice Manual is publically available
from Oct. 9, 2015 ("October/DIWG2015
version"): DRAFT_BPM_FIVE_20151009
Future of Imaging
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Nikon
Sharing
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Conclusions
• Multimedia Forensics is now a
consolidated field but new intriguing
challenges emerge every day.
• The various involved actors are expected:
– To disseminate «basic knowledge»
– To share «best practices» and results
– To identify new challenges
– ..
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Survey
• Matthew C. Stamm, Min Wu and K. J. Ray Liu,
Information Forensics: An Overview of the First
Decade (2013), in: IEEE Access, 1(167-200)
• Alessandro Piva, An Overview on Image Forensics
(2013), in: ISRN Signal Processing, 2013 (Article ID
496701, 22 pages)
- C. Baron - Adobe Photoshop Forensics – Sleuths,
Thruts, and Fauxtography – Thomson Course
Tehcnology - 2009
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
On line ResourcesTutorial by Prof. Hany Farid - Digital Image Forensics:
lecture notes, exercises, and matlab code for a survey
course in digital image and video
forensics. http://www.cs.dartmouth.edu/farid/downloads/tutor
ials/digitalimageforensics.pdf
Special Issue on Multimedia in Forensics, Security and
Intelligence - IEEE Multimedia Magazine Vol. 19, Issue 1,
pp. 17-19, 2012
SOFTWARE: Amped5, Authenticate, Adroit, Four&Six, Izitru,
Ghiro, …
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Credits
• Amped (www.amped.it)
• Undergraduate course: Computer
Forensics – University of Catania
(http://www.dmi.unict.it/~battiato/CF.html)
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Prof. Sebastiano Battiato
Dipartimento di Matematica e Informatica
University of Catania, Italy
Image Processing LAB – http://iplab.dmi.unict.it
Main Contacts
Further Info
Image Processing Lab
Università di Catania
www.dmi.unict.it/~iplab
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015