The instant quick show or hide effect for modal dialogs is good for some scenarios but, for others using a 'transition' is a better and more sophisticated way to open and close the modal window.
If you are creating a modal in Drupal 8 with one line of HTML you can specify dialog options and set fadeIn and fadeOut effect.
<a class="use-ajax btn" data-dialog-type="modal"
data-dialog-options="{"width":1200, "dialogClass": "custom-class-modal", "show": "fadeIn", "hide": "fadeOut"}"
href="/cart">{{ 'Cart'|t }}</a>
Also, you can use jQuery code to set fadeOut effect on closing modal by clicking outside of the modal window.
$(document).on('click', '.ui-widget-overlay.ui-front', function (){
$('.ui-widget-overlay.ui-front').fadeOut(600, function(){ $(this).remove();})
$('.custom-class-modal').fadeOut(600, function(){ $(this).remove();})
});
Branka Dakic