Manipulation Data Collection and Annotation Tool for Media Forensics

Published

June 17, 2019

Author(s)

Eric Robertson, Haiying Guan, Mark Kozak, Yooyoung Lee, Amy Yates, Andrew Delgado, Daniel F. Zhou, Timothée N. Kheyrkhah, Jeff Smith, Jonathan G. Fiscus

Abstract

With the increasing diversity and complexity of media forensics techniques, the evaluation of state-of-the-art detectors are impeded by lacking the metadata and manipulation history ground-truth. This paper presents a novel image/video manipulation Journaling Tool (JT) that automatically or semi-automatically helps a media manipulator record, or journal, the steps, methods, and tools used to manipulate media into a modified form. JT is a unified framework using a directed acyclic graph representation to support: recording the manipulation history (journal); automating the collection of operation-specific localization masks identifying the set of manipulated pixels; integrating annotations and metadata collection; and execution of automated manipulation tools to extend existing journals or automatically build new journals. Using JT to support the 2017 and 2018 Media Forensics Challenge (MFC) evaluations, a large collection of image manipulations was assembled that included a variety of different manipulation operations across image, video, and audio. To date, the MFC's media manipulation team has collected more than 4500 human-manipulated image journals containing over 100,000 images, more than 400 manipulated video journals containing over 4,000 videos, and generated thousands of extended journals and hundreds of auto- manipulated journals. This paper discusses the JT's design philosophy and requirements, localization mask production, automated journal construction tools, and evaluation data derivation from journals for performance evaluation of media forensics applications. JT enriches the metadata collection, provides consistent and detailed annotations, and builds scalable automation tools to produce manipulated media, which enables the research community to better understand the problem domain and the algorithm models.

Proceedings Title

IEEE computer vision and pattern recognition conference 2019

Conference Dates

June 16-20, 2019

Conference Location

Long Beach, CA, US

Pub Type

Conferences

Download Paper

Local Download

Keywords

Media Forensics, Image Annotation, Annotation Toolkit

Citation

Robertson, E. , Guan, H. , Kozak, M. , Lee, Y. , Yates, A. , Delgado, A. , Zhou, D. , Kheyrkhah, T. , Smith, J. and Fiscus, J. (2019), Manipulation Data Collection and Annotation Tool for Media Forensics, IEEE computer vision and pattern recognition conference 2019, Long Beach, CA, US, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=927817 (Accessed January 31, 2025)