Methodology 1
Monday 6 October 2025, 16.00 – 17.30 Beam
Chairs: Tyler Quinn, Benjamin Kendzia
Assessing performance of diverse text mining platforms through INDRA-extracted mechanistic statements
Simone Del Motto (presenter)
Bernice Scholten, Xavier Sa Castro Pinho, Robert van Vorstenbosch, Susan Peters, Benjamin Gyori, Roel Vermeulen
Abstract
INTRODUCTION: The field of Risk Assessment is challenged by the large quantity of literature it needs to process: a huge labor-intensive task. Automated risk-assessment aims to support researchers by providing tools to extract large quantities of information. One of these tools is INDRA (Integrated Network and Dynamical Reasoning Assembler), which makes use of several reading systems (i.e. we selected five according to developer’s scale of usage and efficiency: REACH, EIDOS, SPARSER, ISI, TRIPS), to extract mechanistic information from texts. These are captured in statements, a standardized output format. Currently, accuracy and coverage of the independent reading systems are not well-established. Therefore, in the current study we assess those properties of these reading systems, and evaluated how data-fusion can be applied to further increase their reliabilities.
METHODS: Each reader was run on a manually curated cancer-related ground-truth subset of 12 full-text articles selected for mechanistic richness and variety of content. Next, the set of extracted statements was reviewed for errors of any type (e.g. from entity type- such as proteins, to directions). Moreover, coverage was assessed. Lastly, Random Forest Modelling and Data Fusion were performed to predict errors.
RESULTS: The current pipeline consolidated 1860 mechanistic statements, which have been matched with 1376 manually extracted statements from the same papers. Generally, we observed that the highest number of statements comes from EIDOS (c.a 1000) and REACH (c.a 600). Reach showed lowest false-positive rate. More specific results (i.e. per reading system, per entity type) will be shown.
CONCLUDING: It is crucial to establish the reliability of the INDRA tool to interpret results and enable improvements to the algorithm. The current implementation will be further examined in Lexces2, and the European projects of Endomix and EDC-MASLD.
Creating Directed Acyclic Graphs (DAGs) with domain experts for healthy employment among aging workers
Chantal Goossens (presenter)
Martijn Huisman, Cécile Boot, Astrid de Wind, Mariska van der Horst
Abstract
Background: Directed Acyclic Graphs (DAGs) are visual tools used in epidemiology to represent and clarify assumptions about causal relationships between variables. While they are valuable tools, the quality of DAGs depends on the incorporation of domain expertise. However, structured approaches to meaningfully involve domain experts in the creation of DAGs—especially in complex occupational health contexts—remain underdeveloped.
Objective: This study aims to evaluate a structured, participatory approach to DAG development in collaboration with domain experts. The focus is on understanding how individual vulnerabilities and organizational factors (e.g. sectoral policies) interact to influence healthy and sustained employment among ageing workers.
Methods: The study involves workshops in which domain experts from occupational health, human resources, labour unions, and other areas, collaboratively construct DAGs based on a defined causal question. The process, informed by existing guidance (e.g. Rodrigues et al., 2022), includes brainstorming, refinement, and validation phases. Participants identify key variables, mechanisms, and sources of bias, while building consensus around causal assumptions.
Preliminary Insights: Preliminary insights suggest that the structured, collaborative approach helps uncover both shared assumptions and divergent perspectives across professional backgrounds. Participants found it made implicit causal reasoning more explicit and encouraged reflection on context-specific mechanisms. The process also fostered interdisciplinary dialogue and critical discussions about confounding and data limitations. However, balancing depth and feasibility proved challenging: conversations risked becoming too broad and not manageable. Final DAGs are still in development, but early findings indicate that this method enhances both the relevance of DAGs for applied research and mutual learning among experts.
Conclusion: Although still ongoing, the study demonstrates the potential of expert-guided DAG development to enrich causal modelling in occupational health research. The approach may help bridge the gap between theory, data, and policy by integrating diverse forms of expertise into the analytical design phase.
Small Language model based NAICS 2022 CLassification of Industry from Products and Services: CLIPS
Daniel Russ (presenter)
Daniel E. Russ, Pabitra R. Josse, Jonas S. Almedia, Melissa C. Friesen
Abstract
Objective: Numerous tools are now available that use natural language processing to automatically code free-text job descriptions to standardized occupation classification systems. However, the development of tools for coding industry has received much less attention. Here we describe CLIPS, a tool we developed to identify plausible standardized industry codes based on free-text responses to the question ‘what product is made or services provided.’ This tool can be incorporated into online questionnaires to help participants self-code their industry.
Materials and Methods: CLIPS uses small language models to code free-text industry information to NAICS 2022 codes. It uses a two-step process that first converts (embeds) the text information to numbers and then classifies using a dense classification neural network. Employer name was excluded as an input feature, based on concerns over privacy and the difficulty of obtaining training data. CLIPS provides a score for each of the 689 NAICS 5-digit codes. The industry codes with the highest scores are then included in a list from which the study participant selects the best fit. We validated CLIPS using 1,586 jobs coded to NAICS 2022 by an expert.
Results: Overall, the industry code with the highest score from our preliminary version of CLIPS had a 47.4% agreement with the expert-assigned codes in our validation data set. The CLIPS score predicted agreement with the expert-assigned code. In addition, the expert-assigned code was in the top 3 CLIPS-suggested codes for 66% of the jobs, in the top 6 for 75%, and in the top 10 for 80%.
Conclusion: CLIPS’ ability to identify the expert-assigned code in its highest scoring codes makes it suitable for assisting participants in self-coding their industry. We hope to expand the training data to further improve its performance.
Acknowledgements: This work is funded by the Intramural Research Program of the US National Cancer Institute, Division of Cancer Epidemiology and Genetics.
Structuring quality of working life research in remote and hybrid work: A topic modelling framework for identifying key variables
Sibel Kiran (presenter)
Hakan Orer, Sezer Uguz, Shi (Tracy) Xu, Katharina Fellnhofer, Alexandra Prodromidou, Özge Karanfil, İlker Kayi, Ayse Giz Gulnerman, S Selcuk Surucu, Sibel Sakarya
Abstract
Background: The expansion of remote and hybrid work has reshaped contemporary work environments, creating new opportunities to enhance frameworks for assessing Quality of Working Life (QWL). While existing models offer valuable foundations, further development is needed to reflect evolving dynamics, including psychosocial factors, digital demands, flexibility, and gendered experiences. This study advances a literature-informed, topic modelling approach to develop a standardised set of variables for evaluating QWL in remote and hybrid contexts.
Objectives: To systematically identify key research areas within the QWL literature using topic modelling, and to develop a context-specific, literature-informed assessment instrument comprising a standardised set of core variables for evaluating essential dimensions of QWL in remote and hybrid work settings. Methods: We conducted topic modelling on interdisciplinary publications retrieved from the Scopus® database, using a targeted search strategy applied to titles, abstracts, and keywords. Latent Dirichlet Allocation (LDA) was implemented with the MALLET toolkit in Python, yielding twelve thematic clusters, including systems and design aspects, organisational dynamics, employment conditions, mental health, gender roles, and policy.
Results: The findings informed the development of a multidimensional, standardised instrument comprising variables related to autonomy, work intensity, care responsibilities, digital infrastructure, psychosocial outcomes, and living conditions. It also integrates conceptual dimensions such as ‘pleasure in work,’ ‘mastery,’ and ‘fellowship,’ aligning with occupational health principles and the broader societal values embedded in existing models. This instrument contributes to addressing conceptual gaps in the assessment of remote work, self-employment, and the integration of occupational health and safety (OHS).
Conclusions: By integrating topic modelling with the development of a targeted core set of variables, this study offers a structured approach to support future research and policy discussions on Quality of Working Life (QWL), remote and hybrid work, psychosocial factors, and occupational well-being in digitally mediated work environments.
TNO-Auto occupation coder: A novel, large-language model based multilingual automatic coding tool for occupations
Calvin Ge (presenter)
Xavier SC Pinho, Sadegh M Shahmohammadi
Abstract
Objective Coding to job descriptions to standardised classification systems is an important prerequisite for the application of occupational exposure assessment tools in occupational epidemiology studies. We aimed to create of an automatic job coding tool using large language models (LLMs) capable of handling job description input in different languages and providing output in ISCO-08.
Methods Our approach follows the retrieval-augmented generation technique that modifies responses of LLMs based on a specified set of supplementary information. Our supplementary information included domain knowledge documents defining and exemplifying the ISCO-08 ontology and job coding process, including the ISCO-08 official documentation plus approximately 34,000 ISCO-08 job titles in 28 languages. All text in domain knowledge information and free text input were converted to vector embeddings using Open AI’s “text-embedding-3-large” embedding model. Input and domain knowledge text embeddings were then compared by a retrieval model, which retrieves the 10 ISCO-08 job codes with highest embedding similarity. Finally, the original free text job description input, along with its 10 most similar ISCO job codes, were presented in a prompt to OpenAI’s GTP-4o to select an ISCO-08 job as output.
Results The TNO-Auto occupation coder was applied to approximately 26,000 job descriptions in various European languages for an Eurostat job classification competition. Preliminary results show that our model has the highest adjusted accuracy amongst competing teams at 58%. Further validation of model performance is underway to investigate model performance in different languages and different profession groups.
Conclusion We created an automatic job coding tool capable of accepting multilingual job descriptions as input and providing job codes in ISCO-08 as output. Our general approach may be applied to create automatic coding tools for conversion of multilingual free text input of job or industry descriptions into occupation and industry codes under different classification systems.
The perspective of Dutch healthcare providers on mechanisms underlying burnout symptoms – a complex systems approach
Ruben Heuven (presenter)
Saskia van den Berg, Karin I. Proper, Sandra H. van Oostrom(_
Abstract
Objective: Healthcare providers report burnout complaints significantly more often than other professional groups, leading to consequences such as reduced quality of life and care, high absenteeism and associated costs. Despite broad recognition of this phenomenon, there has been a rising prevalence over the past ten years. Addressing this complex issue requires a shift from linear thinking to a systems perspective—focusing on interaction and feedback. The objective of this study is to explore the mechanisms behind burnout complaints by approaching it as a complex system, from the perspective of healthcare providers.
Material and Methods: Thirteen healthcare providers from eight different professions were recruited through a combination of convenience and snowball sampling. They participated in two Group Model Building sessions to develop a Causal Loop Diagram (CLD). During these sessions, participants identified factors related to living conditions, working conditions, and societal developments that contribute to burnout complaints and indicated their assumed relationships resulting into causal feedback loops.
Results: The CLD contains 24 factors, 55 assumed causal relationships, and 46 unique feedback loops. These were grouped into three overarching themes: one reinforcing and two balancing. The reinforcing theme illustrates how burnout complaints and staff shortages mutually amplify one another through multiple mechanisms – for example, burnout-induced absenteeism can increase workloads, undermine decision latitude and job satisfaction, and thereby perpetuate burnout. The two balancing themes reflect how this vicious cycle is partially offset by supportive leadership, like aligning responsibilities with competences, and rising societal awareness, leading to improvements in working conditions and employment terms.
Conclusion: According to healthcare providers, these complex interacting mechanisms help explain the rising trend in the prevalence of burnout complaints within the sector. Understanding these mechanisms offers valuable insights for developing more effective, system-oriented strategies to address burnout complaints among healthcare providers.