FRAME — Replication Package Replication package for the FRAME paper (ICSME 2026). FRAME turns a mobile UI screenshot into a single embedding by fusing three signals: Visual — CLIP ViT-B/32 on the full screen and on each detected UI element. Textual — on-screen text via Tesseract OCR, embedded with BERT (bert-base-uncased). Structural — a NetworkX graph of UI elements (edges by on-screen distance) with embedding propagation over 1-/2-hop neighborhoods. The fused embedding is evaluated on screen retrieval (MRR / Hits@K). Repository layout config.py # ← single place to set all paths & parameters utils.py # shared helpers UI_Embedding_main.py # core pipeline; makeEmbedding() + graph propagation UIEDComp.py # UIED element detection wrapper imageEmbedding.py # CLIP embeddings (full-screen + per-element) textEmbedding.py # Tesseract OCR -> BERT text embeddings graphCreation.py # build element graph, extract edges embeddingConsolidation.py # Rips-complex / centroid consolidation makeCSV.py # batch: dataset index -> per-screen embeddings CSV embeddingsimilarity.py # retrieval evaluation (MRR / Hits@K) CaseStudyCreate.py # aggregate metrics by screen type t_test.py # paired significance testing DimensionReduction.py # PCA / LLE experiments pca_transformation_matrix.npy # precomputed 140x1792 PCA matrix rips_*.csv # precomputed embedding dumps (see DATA.md) screenshots/ # 3 sample screenshots for the smoke test detectors/Visual/UIED-master/ # vendored UIED detector (third-party, own LICENSE) run_all.sh # end-to-end smoke test on the bundled screenshots Dockerfile # CPU-only, self-contained reproducible image See DATA.md for the embedding dumps' schema and dataset details. Option A — Docker (recommended) Fully self-contained; no manual dependency setup. Requires only Docker. docker build -t frame . docker run --rm frame # runs the smoke test (run_all.sh) docker run --rm -it frame bash # interactive shell to run individual scripts The image bundles the code, data, and pre-downloaded CLIP + BERT weights, so it runs offline. It targets linux/amd64 (some pinned dependencies have no arm64 wheels for Python 3.9); on Apple Silicon it builds and runs under emulation. Option B — Local install Prerequisites Python 3.9.13 (pip 23.3.2) Tesseract OCR system binary — required by pytesseract: macOS: brew install tesseract Debian/Ubuntu: sudo apt-get install tesseract-ocr git — the CLIP dependency is installed from source (see requirements.txt). Network access on first run: clip.load("ViT-B/32") and bert-base-uncased download model weights (~1 GB total) and cache them. Setup python3 -m venv venv source venv/bin/activate pip install -r requirements.txt Configure All paths and parameters live in config.py — edit it once (or override any value via an environment variable of the same name, e.g. EMBEDDINGS_CSV=/data/x.csv python3 embeddingsimilarity.py). Defaults point at the bundled sample data so the smoke test runs with zero edits. Smoke test ./run_all.sh Embeds each screenshot in screenshots/ end-to-end and prints the embedding dimensionality. This does notrequire the labeled dataset. Reproducing the paper The retrieval results run as a pipeline. Important: the committed rips_*.csv dumps reference the original dataset screenshots by absolute path, and the evaluation re-opens those images — so you must supply the dataset (or regenerate the dumps). See the caveat in DATA.md. Generate embeddings from a dataset index CSV (config.EMBEDDING_INDEX_CSV → config.EMBEDDINGS_OUTPUT_CSV): python3 makeCSV.py (Or skip this step and use a bundled rips_*.csv dump directly.) Run retrieval evaluation (config.EMBEDDINGS_CSV → config.RESULTS_OUTPUT_CSV); prints MRR and Hits@{1,5,10}: python3 embeddingsimilarity.py Aggregate metrics by screen type (config.METRICS_INPUT_CSV → config.METRICS_OUTPUT_CSV): python3 CaseStudyCreate.py Significance testing between two result CSVs (paired t-test over MRR / Hits@K): python3 t_test.py Data The four precomputed embedding dumps (RICO + Avgust) and their schema are documented in DATA.md. RICO: https://interactionmining.org/rico. License & citation Code: MIT (see LICENSE). The vendored UIED detector under detectors/Visual/UIED-master/ keeps its own third-party license. Please cite the paper — see CITATION.cff.