Showing 1–14 of 14 results for author: Kun, P

LayerShuffle: Enhancing Robustness in Vision Transformers by Randomizing Layer Execution Order

Authors: Matthias Freiberger, Peter Kun, Anders Sundnes Løvlie, Sebastian Risi

Abstract: Due to their architecture and how they are trained, artificial neural networks are typically not robust toward pruning, replacing, or shuffling layers at test time. However, such properties would be desirable for different applications, such as distributed neural network architectures where the order of execution cannot be guaranteed or parts of the network can fail during inference. In this work,… ▽ More Due to their architecture and how they are trained, artificial neural networks are typically not robust toward pruning, replacing, or shuffling layers at test time. However, such properties would be desirable for different applications, such as distributed neural network architectures where the order of execution cannot be guaranteed or parts of the network can fail during inference. In this work, we address these issues through a number of proposed training approaches for vision transformers whose most important component is randomizing the execution order of attention modules at training time. We show that with our proposed approaches, vision transformers are indeed capable to adapt to arbitrary layer execution orders at test time assuming one tolerates a reduction (about 20\%) in accuracy at the same model size. We also find that our trained models can be randomly merged with each other resulting in functional ("Frankenstein") models without loss of performance compared to the source models. Finally, we layer-prune our models at test time and find that their performance declines gracefully. △ Less

Submitted 5 July, 2024; originally announced July 2024.

AI-generated art perceptions with GenFrame -- an image-generating picture frame

Authors: Peter Kun, Matthias Freiberger, Anders Sundnes Løvlie, Sebastian Risi

Abstract: Image-generation models are changing how we express ourselves in visual art. However, what people think of AI-generated art is still largely unexplored, especially compared to traditional art. In this paper, we present the design of an interactive research product, GenFrame - an image-generating picture frame that appears as a traditional painting but offers the viewer the agency to modify the dep… ▽ More Image-generation models are changing how we express ourselves in visual art. However, what people think of AI-generated art is still largely unexplored, especially compared to traditional art. In this paper, we present the design of an interactive research product, GenFrame - an image-generating picture frame that appears as a traditional painting but offers the viewer the agency to modify the depicted painting. In the current paper, we report on a study where we deployed the GenFrame in a traditional art museum and interviewed visitors about their views on AI art. When provoked by AI-generated art, people need more of the artist's backstory and emotional journey to make the artwork commensurate with traditional art. However, generative AI-enabled interactive experiences open new ways of engaging with art when a turn of a dial can modify art styles or motifs on a painting. A demo can be seen here: https://youtu.be/1rhW4fazaBY. △ Less

Submitted 3 May, 2024; originally announced May 2024.

Comments: Design Research Society conference 2024 (DRS2024), Boston 24-28 June 2024

Algorithmic Ways of Seeing: Using Object Detection to Facilitate Art Exploration

Authors: Louie Søs Meyer, Johanne Engel Aaen, Anitamalina Regitse Tranberg, Peter Kun, Matthias Freiberger, Sebastian Risi, Anders Sundnes Løvlie

Abstract: This Research through Design paper explores how object detection may be applied to a large digital art museum collection to facilitate new ways of encountering and experiencing art. We present the design and evaluation of an interactive application called SMKExplore, which allows users to explore a museum's digital collection of paintings by browsing through objects detected in the images, as a no… ▽ More This Research through Design paper explores how object detection may be applied to a large digital art museum collection to facilitate new ways of encountering and experiencing art. We present the design and evaluation of an interactive application called SMKExplore, which allows users to explore a museum's digital collection of paintings by browsing through objects detected in the images, as a novel form of open-ended exploration. We provide three contributions. First, we show how an object detection pipeline can be integrated into a design process for visual exploration. Second, we present the design and development of an app that enables exploration of an art museum's collection. Third, we offer reflections on future possibilities for museums and HCI researchers to incorporate object detection techniques into the digitalization of museums. △ Less

Submitted 28 March, 2024; originally announced March 2024.

Exploring diversity perceptions in a community through a Q&A chatbot

Authors: Peter Kun, Amalia De Götzen, Miriam Bidoglia, Niels Jørgen Gommesen, George Gaskell

Abstract: While diversity has become a debated issue in design, very little research exists on positive use-cases for diversity beyond scholarly criticism. The current work addresses this gap through the case of a diversity-aware chatbot, exploring what benefits a diversity-aware chatbot could bring to people and how do people interpret diversity when being presented with it. In this paper, we motivate a Q&… ▽ More While diversity has become a debated issue in design, very little research exists on positive use-cases for diversity beyond scholarly criticism. The current work addresses this gap through the case of a diversity-aware chatbot, exploring what benefits a diversity-aware chatbot could bring to people and how do people interpret diversity when being presented with it. In this paper, we motivate a Q&A chatbot as a technology probe and deploy it in two student communities within a study. During the study, we collected contextual data on people's expectations and perceptions when presented with diversity during the study. Our key findings show that people seek out others with shared niche interests, or their search is driven by exploration and inspiration when presented with diversity. Although interacting with chatbots is limited, participants found the engagement novel and interesting to motivate future research. △ Less

Submitted 13 February, 2024; originally announced February 2024.

Comments: Design Research Society conference 2022, Bilbao, 25 June - 3 July, 2022

Fooling Contrastive Language-Image Pre-trained Models with CLIPMasterPrints

Authors: Matthias Freiberger, Peter Kun, Christian Igel, Anders Sundnes Løvlie, Sebastian Risi

Abstract: Models leveraging both visual and textual data such as Contrastive Language-Image Pre-training (CLIP), are the backbone of many recent advances in artificial intelligence. In this work, we show that despite their versatility, such models are vulnerable to what we refer to as fooling master images. Fooling master images are capable of maximizing the confidence score of a CLIP model for a significan… ▽ More Models leveraging both visual and textual data such as Contrastive Language-Image Pre-training (CLIP), are the backbone of many recent advances in artificial intelligence. In this work, we show that despite their versatility, such models are vulnerable to what we refer to as fooling master images. Fooling master images are capable of maximizing the confidence score of a CLIP model for a significant number of widely varying prompts, while being either unrecognizable or unrelated to the attacked prompts for humans. The existence of such images is problematic as it could be used by bad actors to maliciously interfere with CLIP-trained image retrieval models in production with comparably small effort as a single image can attack many different prompts. We demonstrate how fooling master images for CLIP (CLIPMasterPrints) can be mined using stochastic gradient descent, projected gradient descent, or blackbox optimization. Contrary to many common adversarial attacks, the blackbox optimization approach allows us to mine CLIPMasterPrints even when the weights of the model are not accessible. We investigate the properties of the mined images, and find that images trained on a small number of image captions generalize to a much larger number of semantically related captions. We evaluate possible mitigation strategies, where we increase the robustness of the model and introduce an approach to automatically detect CLIPMasterPrints to sanitize the input of vulnerable models. Finally, we find that vulnerability to CLIPMasterPrints is related to a modality gap in contrastive pre-trained multi-modal networks. Code available at https://github.com/matfrei/CLIPMasterPrints. △ Less

Submitted 16 April, 2024; v1 submitted 7 July, 2023; originally announced July 2023.

Comments: This work was supported by a research grant (40575) from VILLUM FONDEN

Complex Daily Activities, Country-Level Diversity, and Smartphone Sensing: A Study in Denmark, Italy, Mongolia, Paraguay, and UK

Authors: Karim Assi, Lakmal Meegahapola, William Droz, Peter Kun, Amalia de Gotzen, Miriam Bidoglia, Sally Stares, George Gaskell, Altangerel Chagnaa, Amarsanaa Ganbold, Tsolmon Zundui, Carlo Caprini, Daniele Miorandi, Alethia Hume, Jose Luis Zarza, Luca Cernuzzi, Ivano Bison, Marcelo Dario Rodas Britez, Matteo Busso, Ronald Chenu-Abente, Fausto Giunchiglia, Daniel Gatica-Perez

Abstract: Smartphones enable understanding human behavior with activity recognition to support people's daily lives. Prior studies focused on using inertial sensors to detect simple activities (sitting, walking, running, etc.) and were mostly conducted in homogeneous populations within a country. However, people are more sedentary in the post-pandemic world with the prevalence of remote/hybrid work/study se… ▽ More Smartphones enable understanding human behavior with activity recognition to support people's daily lives. Prior studies focused on using inertial sensors to detect simple activities (sitting, walking, running, etc.) and were mostly conducted in homogeneous populations within a country. However, people are more sedentary in the post-pandemic world with the prevalence of remote/hybrid work/study settings, making detecting simple activities less meaningful for context-aware applications. Hence, the understanding of (i) how multimodal smartphone sensors and machine learning models could be used to detect complex daily activities that can better inform about people's daily lives and (ii) how models generalize to unseen countries, is limited. We analyzed in-the-wild smartphone data and over 216K self-reports from 637 college students in five countries (Italy, Mongolia, UK, Denmark, Paraguay). Then, we defined a 12-class complex daily activity recognition task and evaluated the performance with different approaches. We found that even though the generic multi-country approach provided an AUROC of 0.70, the country-specific approach performed better with AUROC scores in [0.79-0.89]. We believe that research along the lines of diversity awareness is fundamental for advancing human behavior understanding through smartphones and machine learning, for more real-world utility across countries. △ Less

Submitted 16 February, 2023; originally announced February 2023.

Comments: ACM CHI 2023

Generalization and Personalization of Mobile Sensing-Based Mood Inference Models: An Analysis of College Students in Eight Countries

Authors: Lakmal Meegahapola, William Droz, Peter Kun, Amalia de Gotzen, Chaitanya Nutakki, Shyam Diwakar, Salvador Ruiz Correa, Donglei Song, Hao Xu, Miriam Bidoglia, George Gaskell, Altangerel Chagnaa, Amarsanaa Ganbold, Tsolmon Zundui, Carlo Caprini, Daniele Miorandi, Alethia Hume, Jose Luis Zarza, Luca Cernuzzi, Ivano Bison, Marcelo Rodas Britez, Matteo Busso, Ronald Chenu-Abente, Can Gunel, Fausto Giunchiglia , et al. (2 additional authors not shown)

Abstract: Mood inference with mobile sensing data has been studied in ubicomp literature over the last decade. This inference enables context-aware and personalized user experiences in general mobile apps and valuable feedback and interventions in mobile health apps. However, even though model generalization issues have been highlighted in many studies, the focus has always been on improving the accuracies… ▽ More Mood inference with mobile sensing data has been studied in ubicomp literature over the last decade. This inference enables context-aware and personalized user experiences in general mobile apps and valuable feedback and interventions in mobile health apps. However, even though model generalization issues have been highlighted in many studies, the focus has always been on improving the accuracies of models using different sensing modalities and machine learning techniques, with datasets collected in homogeneous populations. In contrast, less attention has been given to studying the performance of mood inference models to assess whether models generalize to new countries. In this study, we collected a mobile sensing dataset with 329K self-reports from 678 participants in eight countries (China, Denmark, India, Italy, Mexico, Mongolia, Paraguay, UK) to assess the effect of geographical diversity on mood inference models. We define and evaluate country-specific (trained and tested within a country), continent-specific (trained and tested within a continent), country-agnostic (tested on a country not seen on training data), and multi-country (trained and tested with multiple countries) approaches trained on sensor data for two mood inference tasks with population-level (non-personalized) and hybrid (partially personalized) models. We show that partially personalized country-specific models perform the best yielding area under the receiver operating characteristic curve (AUROC) scores of the range 0.78-0.98 for two-class (negative vs. positive valence) and 0.76-0.94 for three-class (negative vs. neutral vs. positive valence) inference. Overall, we uncover generalization issues of mood inference models to new countries and how the geographical similarity of countries might impact mood inference. △ Less

Submitted 5 November, 2022; originally announced November 2022.

Comments: ACM IMWUT 2022, To be presented at ACM Ubicomp 2023

Observation of competing, correlated ground states in the flat band of rhombohedral graphite

Authors: Imre Hagymási, Mohammad Syahid Mohd Isa, Zoltán Tajkov, Krisztián Márity, Oroszlány László, János Koltai, Assem Alassaf, Péter Kun, Konrád Kandrai, András Pálinkás, Péter Vancsó, Levente Tapasztó, Péter Nemes-Incze

Abstract: In crystalline solids the interactions of charge and spin can result in a variety of emergent quantum ground states, especially in partially filled, topological flat bands such as Landau levels or in 'magic-angle' bilayer graphene. Much less explored is rhombohedral graphite (RG), perhaps the simplest and structurally most perfect condensed matter system to host a flat band protected by symmetry.… ▽ More In crystalline solids the interactions of charge and spin can result in a variety of emergent quantum ground states, especially in partially filled, topological flat bands such as Landau levels or in 'magic-angle' bilayer graphene. Much less explored is rhombohedral graphite (RG), perhaps the simplest and structurally most perfect condensed matter system to host a flat band protected by symmetry. By scanning tunneling microscopy we map the flat band charge density of 8, 10 and 17 layers and identify a domain structure emerging from a competition between a sublattice antiferromagnetic insulator and a gapless correlated paramagnet. Our density-matrix renormalization group calculations explain the observed features and demonstrate that the correlations are fundamentally different from graphene based magnetism identified until now, forming the ground state of a quantum magnet. Our work establishes RG as a new platform to study many-body interactions beyond the mean-field approach, where quantum fluctuations and entanglement dominate. △ Less

Submitted 15 July, 2022; v1 submitted 26 January, 2022; originally announced January 2022.

Comments: supplementary information included

Journal ref: Science Advances 8, eabo6879 (2022)

Robust quantum point contact operation of narrow graphene constrictions patterned by AFM cleavage lithography

Authors: Péter Kun, Bálint Fülöp, Gergely Dobrik, Péter Nemes-Incze, István Endre Lukács, Szabolcs Csonka, Chanyong Hwang, Levente Tapasztó

Abstract: Detecting conductance quantization in graphene nanostructures turned out more challenging than expected. The observation of well-defined conductance plateaus through graphene nanoconstrictions so far has only been accessible in the highest quality suspended or h-BN encapsulated devices. However, reaching low conductance quanta in zero magnetic field, is a delicate task even with such ultra-high mo… ▽ More Detecting conductance quantization in graphene nanostructures turned out more challenging than expected. The observation of well-defined conductance plateaus through graphene nanoconstrictions so far has only been accessible in the highest quality suspended or h-BN encapsulated devices. However, reaching low conductance quanta in zero magnetic field, is a delicate task even with such ultra-high mobility devices. Here, we demonstrate a simple AFM-based nanopatterning technique for defining graphene constrictions with high precision (down to 10 nm width) and reduced edge-roughness (+/- 1 nm). The patterning process is based on the in-plane mechanical cleavage of graphene by the AFM tip, along its high symmetry crystallographic directions. As-defined, narrow graphene constrictions with improved edge quality enable an unprecedentedly robust QPC operation, allowing the observation of conductance quantization even on standard $SiO_2/Si$ substrates, down to low conductance quanta. Conductance plateaus, were observed at $ne^2/h$, evenly spaced by $2e^2/h$ (corresponding to n = 3, 5, 7, 9, 11) in the absence of an external magnetic field, while spaced by $e^2/h$ (n = 1, 2, 3, 4, 5, 6) in 8T magnetic field. △ Less

Submitted 8 October, 2020; originally announced October 2020.

Comments: Main paper and supplementary informations

In situ tuning of symmetry-breaking induced non-reciprocity in giant-Rashba semiconductor BiTeBr

Authors: Mátyás Kocsis, Oleksandr Zheliuk, Péter Makk, Endre Tóvári, Péter Kun, Oleg Evgenevich Tereshchenko, Konstantin Aleksandrovich Kokh, Takashi Taniguchi, Kenji Watanabe, Justin Ye, Szabolcs Csonka

Abstract: Non-reciprocal transport, where the left to right flowing current differs from the right to left flowing one, is an unexpected phenomenon in bulk crystals. BiTeBr is a non-centrosymmetric material, with a giant Rashba spin-orbit coupling which presents this unusual effect when placed in an in-plane magnetic field. It has been shown that this effect depends strongly on the carrier density, however,… ▽ More Non-reciprocal transport, where the left to right flowing current differs from the right to left flowing one, is an unexpected phenomenon in bulk crystals. BiTeBr is a non-centrosymmetric material, with a giant Rashba spin-orbit coupling which presents this unusual effect when placed in an in-plane magnetic field. It has been shown that this effect depends strongly on the carrier density, however, in-situ tuning has not yet been demonstrated. We developed a method where thin BiTeBr flakes are gate tuned via ionic-liquid gating through a thin protective hBN layer. Tuning the carrier density allows a more than \SI{400}{\percent} variation of the non-reciprocal response. Our study serves as a milestone on how a few-atomic-layer-thin van der Waals protection layer allows ionic gating of chemically sensitive, exotic nanocrystals. △ Less

Submitted 17 August, 2020; v1 submitted 13 August, 2020; originally announced August 2020.

Journal ref: Phys. Rev. Research 3, 033253 (2021)

Dynamic strain in gold nanoparticle supported graphene induced by focused laser irradiation

Authors: András Pálinkás, Péter Kun, Antal A. Koós, Zoltán Osváth

Abstract: Graphene on noble-metal nanostructures constitutes an attractive nanocomposite with possible applications in sensors or energy conversion. In this work we study the properties of hybrid graphene/gold nanoparticle structures by Raman spectroscopy and Scanning Probe Methods. The nanoparticles (NPs) were prepared by local annealing of gold thin films using focused laser beam. The method resulted in a… ▽ More Graphene on noble-metal nanostructures constitutes an attractive nanocomposite with possible applications in sensors or energy conversion. In this work we study the properties of hybrid graphene/gold nanoparticle structures by Raman spectroscopy and Scanning Probe Methods. The nanoparticles (NPs) were prepared by local annealing of gold thin films using focused laser beam. The method resulted in a patterned surface, with NPs formed at arbitrarily chosen microscale areas. Graphene grown by chemical vapour deposition was transferred onto the prepared, closely spaced gold NPs. While we found that successive higher intensity (6 mW) laser irradiation increased gradually the doping and the defect concentration in SiO2 supported graphene, the same irradiation procedure did not induce such irreversible effects in the graphene supported by gold NPs. Moreover, the laser irradiation induced dynamic hydrostatic strain in the graphene on Au NPs, which turned out to be completely reversible. These results can have implications in the development of graphene/plasmonic nanoparticle based high temperature sensors operating in dynamic regimes. △ Less

Submitted 12 June, 2018; originally announced June 2018.

Comments: 18 pages, 5 figures

Large intravalley scattering due to pseudo-magnetic fields in crumpled graphene

Authors: Péter Kun, Gergő Kukucska, Gergely Dobrik, János Koltai, Jenő Kürti, László P. Biró, Levente Tapasztó, Péter Nemes-Incze

Abstract: The pseudo-magnetic field generated by mechanical strain in graphene can have dramatic consequences on the behavior of electrons and holes. Here we show that pseudo-magnetic field fluctuations present in crumpled graphene can induce significant intravalley scattering of charge carriers. We detect this by measuring the confocal Raman spectra of crumpled areas, where we observe an increase of the D'… ▽ More The pseudo-magnetic field generated by mechanical strain in graphene can have dramatic consequences on the behavior of electrons and holes. Here we show that pseudo-magnetic field fluctuations present in crumpled graphene can induce significant intravalley scattering of charge carriers. We detect this by measuring the confocal Raman spectra of crumpled areas, where we observe an increase of the D'/D peak intensity ratio by up to a factor of 300. We reproduce our observations by numerical calculation of the double resonant Raman spectra and interpret the results as experimental evidence of the phase shift suffered by Dirac charge carriers in the presence of a pseudo-magnetic field. This lifts the restriction on complete intravalley backscattering of Dirac fermions. △ Less

Submitted 26 February, 2019; v1 submitted 26 January, 2018; originally announced January 2018.

Journal ref: npj 2D Materials and Applications (2019)

Exfoliation of single layer BiTeI flakes

Authors: Bálint Fülöp, Zoltán Tajkov, János Pető, Péter Kun, János Koltai, László Oroszlány, Endre Tóvári, Hiroshi Murakawa, Yoshinori Tokura, Sándor Bordács, Levente Tapasztó, Szabolcs Csonka

Abstract: Spin orbit interaction can be strongly boosted when a heavy element is embedded into an inversion asymmetric crystal field. A simple structure to realize this concept in a 2D crystal contains three atomic layers, a middle one built up from heavy elements generating strong atomic spin-orbit interaction and two neighboring atomic layers with different electron negativity. BiTeI is a promising candid… ▽ More Spin orbit interaction can be strongly boosted when a heavy element is embedded into an inversion asymmetric crystal field. A simple structure to realize this concept in a 2D crystal contains three atomic layers, a middle one built up from heavy elements generating strong atomic spin-orbit interaction and two neighboring atomic layers with different electron negativity. BiTeI is a promising candidate for such a 2D crystal, since it contains heavy Bi layer between Te and I layers. Recently the bulk form of BiTeI attracted considerable attention due to its giant Rashba interaction, however, 2D form of this crystal was not yet created. In this work we report the first exfoliation of single layer BiTeI using a recently developed exfoliation technique on stripped gold. Our combined scanning probe studies and first principles calculations show that SL BiTeI flakes with sizes of 100 $μみゅー$m were achieved which are stable at ambient conditions. The giant Rashba splitting and spin-momentum locking of this new member of 2D crystals open the way towards novel spintronic applications and synthetic topological heterostructures. △ Less

Submitted 27 September, 2017; originally announced September 2017.

Comments: 20 pages, 5 figures

Journal ref: 2D Mater. 5 (2018) 031013

Highly wear-resistant and low-friction Si3N4 composites by addition of graphene nanoplatelets approaching the 2D limit

Authors: Orsolya Tapaszto, Jan Balko, Viktor Puchy, Peter Kun, Gergely Dobrik, Zsolt Fogarassy, Zsolt Endre Horvath, Jan Dusza, Katalin Balazsi, Csaba Balazsi, Levente Tapaszto

Abstract: Graphene nanoplatelets (GNPs) have emerged as one of the most promising filler materials for improving the tribological performance of ceramic composites due to their outstanding solid lubricant properties as well as mechanical and thermal stability. Yet, the addition of GNPs has so far provided only a very limited improvement in the tribological properties of ceramics, particularly concerning the… ▽ More Graphene nanoplatelets (GNPs) have emerged as one of the most promising filler materials for improving the tribological performance of ceramic composites due to their outstanding solid lubricant properties as well as mechanical and thermal stability. Yet, the addition of GNPs has so far provided only a very limited improvement in the tribological properties of ceramics, particularly concerning the reduction of their friction coefficient. This is most likely due to the challenges of achieving a lubricating and protecting tribo-film through a high GNP coverage of the exposed surfaces. Here we show that this can be achieved by efficiently increasing the exfoliation degree of GNPs down to the few-layer (FL) range. By employing FL-GNPs as filler material, the wear resistance of Si3N4 composites can be increased by about twenty times, the friction coefficient reduced to nearly its half, while the other mechanical properties are also preserved or improved. Using confocal Raman microscopy, we were able to demonstrate the formation of a continuous FL- GNP tribo-film, already at 5wt% FL-GNP content. △ Less

Submitted 10 February, 2017; originally announced February 2017.